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ABSTBACT 

In conpai^son vith traditional experlaental design, 
which is concerned vith i^ibat happened, a case study appi;.oacii is more 
appropriate for answering the question of why or how7 soiethlng 
happened. As an altenativjs 

co»ple»entaTy-vocatio1ial-ed'ucatlon-evaluation apprbach, the case 
stud] attcapts to describe and analyze scne prograa in coaprehensive 
teraa with its idiosyncracies and conplexitles, frequently as it 
unfolds over tlae, case studies have advantages«-opp9rtuQitles ' to 
forvulate prckleas as the evaluatlcn progresses, aodify data 
categorlM Gcnittantly, and reduce distortion of variables— and 
lialtatlonib-^need for special training In conducting then and careful 
definiticn of ^ case, their length, and their generalizabllity. Twelve 
tteps coaprise the three stages in the case study process: 
pre*fiiild«ork: fieldwork: and analysis, verfication, and synthesis. 
The preliainary step^ involved in case study design are setting 
boundaries, defining the unit of analysis, selecting a site, 
establishing int.tl«^l contacts, developing data collection^ systens, 
and defining fl^iwwotk procedures* The field work stage involves 
»tt0l training, logistics of field operations, and data collection 
iislng qt^alitatlve aethods (interviewing, observing, and gathering 
dita unokstruslvelf)* Steps in the final 'stages-analysis, 
verification, and synthesis— include analyzing data, reporting the 
llndlngs, and utilizing the findings^ (A case study puzzle and 
Annotated fclkllography are appended.) (TIB) 

* Bepcodvctlons suppil-ed by EDB6 arc the best that can be lade 
;# frca the original- document. 
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. Evaluation activities have been criticized for not widely being used to improve programs and 
policies in education and other human resource fields. These criticisms stem from many sources 
including the rdattzatton fhatlhe evatualion questions being raised oft^ are not answered by using 
the traditional evaluation methodologies. As a result, there has been a turn to searching for other 
evaluation methodologies to complement the ones curreptly^in use in an effort to produce evaluation 
findings that are more useful to program and policy level decision- makers. One such methodology is 
the case study. This trend is reflected in-journals such as the Review of Educetional Res^rch, 
Amen'cen Educational Research Journal, Educational Research^, Harvard Education Review, the 
Administrative Science Quarterly, Evaluation Review, and Anthropology and Education Quarterly, 
which have all published case study manuscripts. 

' This handbook is designed to address these concerns for vocational education. It is organized 
to provide the reader with an understanding of cW study methods, its strengths, Jimitations, and 
how to conduct case studies. 

The National Center expresses its appreciation to the many individuals who contributed to this 
handbook. Special thanks are extended to Janet E. Spirer, who prepared the handbook and directed 
the project. Sh&was assisted by Ron Schilling, graduate research associate, who prepared the annotated 
bibliography, and Nancy R Stephens, program assistant. Recognition is given to N. L. McCaslin, 
associate director for evaluation and policy, and Floyd McKinney, program direotor, who provided 
invaluable assistance throughout the project. In addition, several persons provided suggestions on the 
structure of the handbook in its preliminary stages including Michael Q. Patton, University of 
Minnesota; Ray Rist, Cornell University; and Robert Stake, University of Illinois. .Robert Bogdan, 
Syracuse UniversHy, Yvonne S. Lincoln, University of Kansas, Paul A. Pohland, University of New 
Mexico, and Deborah Coleman and Michael Crowe, both of the National Center, are to be acknowl- 
edged Yor their caroful ancf thoughtful review of an early draft of the manuscript. Their suggestions 
were often incorpo^ted into this final manuscript. 



A special npte of appreciation is extended to Nancy Powell, who typed the manuscript, and 
Carolyn Hamilton, who provided editorial assistance. ^ 



Ob bihalf of the National Center, \ want to express appreciation to the Bureau of Occupational, 
and Atlult |dgcation, U.S. Office of Education, for sponsoring this evaluation handbook. 



Robert E. Taylor 
Executive Director^ 
The National Center for Research in 
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lC CHAPTER I 

INTRODUCTION 

Be a good craftsman. Avoid a rigid set of proceduces. Above all seek to 
develop and to use the sociological imagination. Avoid fetishism of method 
and technique. Urge the rehabilitation of the unpretentious intellectual 
craftsman, and try to become a craftsman yourself., Let every man be his 
own methodologist ... 

C. Wright Mills 

The Sociological Imagination ^ 



Over twenty years agp, C. Wright Mills warned against the tendency to transform the appropriate 
' Into the orthodox in social research by castigating researchers who have become aligned with one 
research method or technique to the ex6lusion of all others. When this happens, often they become 
so enamored of one method or technique that it becomes an end unto itself. However, "the refusal to 
fecogilize that there are different ways of 'knowing' does no^ mean they do not exist."^ 

Mills encourages the development of the "sociological Imagination" as a means to raise questions 
whereby the evaluator has the cajaacity to shift f rom ont perspective to another (e.g., tingle student 
to^classroom, pollttcal factors to' personal factors, persoifil biases to group biases). The sociological 
imagination In each of us is developed over time. It Is n<^ possible to compile a handbook on how to 
develop it. This handbook does provide a framework for how to conduct an evaluation that was 
spawned from the sociological imagination, using the case study approach. 



'Purpose of the Handbook 

the purpose of this handbook is to introduce vocational educators to the case study method 
as a means by which programs may be better understo<f) and policies developed which ultimately 
provide improved delivery of services. Concern over^ountability of public funds is )-eflected in 
increased federal, state, and local evaluation requirements. Simultaneously, the limitations of 
expej-imental designs to adequately evaluate vocational education has led educators to search for r 
atterniitive evaluation methods. One alternative approach is the case study, which ciraws on work in 
qualltfttive methodology, phenomenology, symbolic interactionism, Gestalt psychology, and ethno- 
methodology. It may be defined as an intensive, detailed analysis and description of a tingle organism, 
institution, or phenomenon in the context of Its environment.' In other words, it is a way to describe 
and analyze, for example, some vocational educatioYi program, school, or policy in comprehensive 
terms With its idiosyncracies and in its complexity, often as It unfolds over time. 

' This htndkK)Oi( 9\m to introduce the reeder to an alt§rnatl\/e approach to evaluation. It if not 
the intent to berate traditional experimental design. Rather, it is based on the premise that while 
tome 0vaiuation°quettiont may be answered best by traditional experimental design, others have 
been raited that the traditional experimental design has not answered adequately. ,For example, the 



firtttwo handbooks In this Evaluation Handbook Series^ deal with hpw to conduct folloyv-up . 
studies of former vocational education students. However, the types of follow-up studies represented 
In these handbooks are quantitative in nature; that is, they primarily describe what happened but 
not why, or how, it happened. To-answer the latter evaluation question, tho case study method is 
more appropriate. 



The Audi«nc0 



In sum, the purpose of this handbook is to intr(>duce the case study method to the reader as an 
alternative approach to asking evaluation questions and collecting evaluation information. Not only 
is the case study method useful mils own right (i.e., by providing evaluation information not currently 
available), it'is also complementary to ongoing evaluation activities in local vocational education" 
programs, and in state and local education agencies. Teacher educators and students at both the 
undergraduate and graduate levels might also find this handbook to be informative and useful. 

Structure of the Handbook 

The text is divided into three major parts which were selected 'to address two prevalent criticisms 
of evaluation efforts: the lack of policy relevance and the lack^f methodological rigor. 

i ^ 

Briefly, Part One is focused on the process of evaluation and how, wh6re, and why the case \ 
study method fits. It 

• provides the reader with a short overview of the evaluation process and the prevalent 
criticisms found in the literature; 

• introduces the case study as an Alternative, yet complementary, evaluation approach 
which finds its roots in the naturalistic inquiry mode; and 

• discusses quel it^ve data collection methods. ' 

After completing Part One, it is anticipated that the reader will have a solid foundation in case study 
methods upon which to build further skills and expertise through the readings cited in the selected 
annotated bibliography (Chapter IX). 

Part Two of the handbook differs f r^om Part One in that it is addressed to the practice of the 
case study method. It is designed to "walk through" each of the three stages of doing a case study. 
Each stage is divided into steps. Each step is accompanied by wmple forms and procedures when 
appi icable and worksheets to help the reader design a case study. 

y 

Part Three consists of a summary and a selected annotated bibliography to help the reader find 
additional materials that may be of interest. 

Souroet of Information ' ^ 

The Information contained In this handbook is drawn from two sources. First, the experience* 
of othen as de«:rlbed In the literature have been reviewed and synthesized. Second, the case study 
experiences of the author-both with vocational education and Comprehensive Employment and 
Training Act (CETA) programs-and the evaluation and policy division of the National Center for 



Research in Vocational Education have been used as a foundation for the materials contained in 
Part Two. Briefly, these experiences include case studies of factors affecting changes in placement 
rates of vocational education programs, iiiterpreting outcome medsur^ of vocational education 
programs, arid the effectiveness of CETA service delivery under Titles j J I, III, and VI. 

Case Study Knowledge Prof ile 

before you beqin reading, take a few minutes to think about the case study method and to 
determine your present level of knowledge by completing the Case Study Profile (Figure 1 ). This 
profile is designed to help you: 

• think through your evaluation needs; 

• become familiar with evaluation issues you may not haveH:onsidered; 
e gain an overview of the contents of this handbook; and 

• decide which sections of the handbook you should study in dep,th, read casually, or skim.^ 

To complete the case study knowledge profile, rate your current level of knowledgeS^f each 
question by darkening the appropriate circle in the left-hand columns. The pabe numbers on the 
right identify the location of th^ information in the text that is helpful in addressing the questioj;). 
An example appears below: 
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CHAPTER II 
THE EVALUATION PROCESS 



In order to place the case iSidy in perspective. Chapter 1 1 briefly reviews the evaluation 
proven before a natUratfstIc approach to evaluation Is Introduced. This, in turn, should provldV a 
framtwork for undenrtandihg the cate^ study method. 



WhyEvaluatt? 

*■ ' 

Evaluatiqn is sotrlething we do every day-^froo) the toothpaste we choose to purchase to th^ 
movies v»e d^ii<te to see. We use a more systematized evaluation proqess to hriake choices' betvi^een] 
for example, whlbl^ product brands we a^ould purchase or which programs we sh6uld continu^o^ 
fund, as well as to improve different products or activities. 
> • ^ -r' ' . .. ' 

The educational syst^ is no exception U|hen it comes to conducting evaluations. Throughoui 
the centuries teachers and other interested persons have done their share of evaluating, whether it 
was student performance, textbooks, or other kinds of instructional materials. More formal evalua- 
tion efforts may be traced back thousands of years to the Chinese ^ivil service examinations. 

* ' ■ ' ' ■ 

Durins the sixties and seventies, added Importance was placed on evaluation in response to the 
political and eponomic milieu as illustrated by: . I 

• The prolrferation of Qreat Society programs in the sixties resulted in a substantial amount 
of resources being expended vvithout a clear deterrpi nation of hpw the programs were 
being implemented or th^ir overall effectiveness. 

• The need to allocate scarce resources among alternatlvts came to the fore in the seventies 
with the advent of escaliiting inflation rates and other economic prohjenxs. As taxpayers 
began to express feelings of being pushed to the Mmit (e.g., the passage of Propositiori 13 
In California), and the call for accountability of public expenditures emerged. 

Purpose of Evaluation ' ^ | 

'• ' . * * ' ' ■ ' ■■ • 

While-these two trends led to increased visibility for evaluation, the evaluation community was 
trying to defir^e the evaluation process and its purpose. In fact, evaluation was (an< is) generally 
perceived as being in flux and transition. 'As a result, several terms Kave been used as synonyms for 
evaluation over the yeirs-meakurement, assessment, appraisal, auditing, grading-and over fifty 
diffarant evaluation models have been proposed.^ However, many of the evaluation models draw 
upon tfiese purpoita of evaluation identified by Anderson and Ball: ^ 

To contribute to d<K;tilon8 about program installation; 
« To contribute to decisions about program contihuation, expansion, or certification; 



• To contribute to decisions about program modification; 

• To obtain evidence to Jallv support fofr a program; 

To obtain evidence-t<5 rally opposition to a program; and 

• To contribute to the understanding of basic psychological, social, and other processes. 



What is Evaluation? 

This handbook subscribes to the idea that evaluation is a process, not a specific procedure. 
Thus, it may use many different models and specific procedures depending upon the questions to 
be answered and the information needs of the audience. Evaluation is more than objectives-based. 
While the attainment of objectives is not ignored, evaluation recognizes that programs haue unaruici-^ 
pated outcomes as well. At times, the value or harm of these unanticipated outcomes may be equal 
to or more important than the objectives. Evaluation is more than evaluating the results of fi 
program. That is, evaluation can focus on the dynamics of a program while the prograna is In progress. 
Studying the different processes involved in programming (e.g., planning, implementation) at each 
of the key stages is essential to program success. Evaluation is more than instructional evaluation. 
It can also be concerned with, for example, establishing priorities among instructional prdgrams, the 
aHocation of resburces, and so forth*. Finally, evaluation is a management tool. It provides ti basis ^ 
for making program ctecisions and for responding*to needed program changes. To pariiphraie the 
Phi Delta Kappa definition, evaluation is a process to delineate, obtain, and provide useful Informa- 
tion for judging decision alternatives.' Also, the evaluator has a responsibility tb help decislon^makers 
use the evaluation information. \ 

\ 

Shortcomings with Evaluation . \ 

■ • * . 

While stresi has been placed upon the need for the type of information evaluation can provide; 
a review of the literature® reveals that evaluation efforts to date have been inadequate. The literature 
Is replete with reasons for the failure of evaluation efforts. They include the following: 

• Since evaluation occurs in an action setting, many questions have been raised about using 
experimental designs in evaluation activities. For example, the randomization of subjects 
is often difficult, and it may be inequitable to deny services to eligible participants in 
order to maintain a randomised experiment.® In addition, control groups are often difficult 
to obtain. . 

» • Programs often have unclear anb/or multiple objectives stressing individualized service , 
delivery.^^ This is especially true with vocational education progrdnis, which implement a 
variety of activities and sen/ices covering a multitude of program objectives (e.g.*. Increasing 
skill levels, providing work experiences. Increasing p^uctivity of human resources) which 
may be competitive. / 

• The imposition of artificial and arbitrary restraints on the scope of ^ study may lead to 
tiSe neglect of other Informatfon salient to the question of how programs are implemented. 

. • Large samples needed to make Itatlstical generalizations may lead tp insensltiyity to local 
perturbations and unbsual efforts, since typical results are seldom studied in detail. 

These shortcomings have been summarized by Crowe and Adarfis in reference to assessing 
experiential education pi-ogrami: 



Qlatiical rtMarch paradigms may have limited utility for testing the effects of 
experiential education programis, especially in the, formative developmental 
stages. While classical research can provide infornVotion to individuals, tt)p 
conditions appear to be such that experiential education is not ready for this 
.In depth examination. The emphasis on using classical research paraditJtns in 
^ evaluating these programs can lead to experiments that are elegantly designed 
but oft#i limited in ibdpe and rolevance., . . The rigidity of the experimental 
s model may, \t\ fact, perforn) a disservice ih educational evaluatione.s^^^ 

A.Naturalistic Approach 

' At this4:>oint, the reader may wonder^how mandated evaluation requirements can |>e met if 
many evaluation efforts have been determined to be inadequate. This senSe of confusion reflects 
the currant state of evaluation efforts. Although the primary strength of the naturalistic approach 
has often been perceived by the social sciences aaJts^tility in hypothesis generation, it is the richness 
of information provided by these naturalistic metliodologies that is leading them again to the fore 
in educational evaluation. These methods have traditionally been used in professions such as law, 
medicine, and journalism, and h^vd been ceritral tc the disciplines of anthropology and sociology. 
Thus, several different evaluation niodels have been developed and are currently being tested, some 
of which art based J^n naturalistic approaches.i^^ ^ \ , 

As fVientioned before, the intention-of this handbook is not to berate the traditional experimental 
methodology. It is still very much a part o^thiB* evaluation picture and provides useful information on 
some^kinds of questions. Rather, the point is that alternatives are available. Hamilton et al.^^ and 
Patton^® note that the application of the traditional experimental approach to evali^ation of. programs 
is often a cumbersome and inadequate procedure. As Hamilton e|t al. state: 

We are not, of course, arguing here against the use of experiinental, longitudinal, 
or survey research methods as such. Rather, for the reasons suic/gested, we submit 
that they are ifsually inappropriate, ineffective^ or insufficient for progranfi evalu- 
ation purposes. , 

Several alternative paradigms have been offered. Underlying each akernative paradigm is a 
commitment to naturalism, that it/ behavioral specimens are collected that "reflect the actual temporal 
sequence of the behaS^ior under analysis and . . . show how each interactant influenced and Was influ- 
enced by all others in the ^behavioral situation." The npturalistic approabh views the program from, 
a holistic; inductive persp^tive, drawing on qualitative ifiethociology, Gesitalt psychology, phenome- 
nology, and ethnomethodology. Thus, it uses methods for "getting inside" complex situations so that 
the researcher can conrfe tolindel^tanci the situation as it Is understood by the participants. Through 
understanding the points of view of the various actors in a social situation, ttfe dynamics of the social 
processes at work may become unlocked and visible. ' ' 

. This handbook provider guidelines and apiMi^tior^s for one naturalistic approach: the case 
study* The case study is ndt a siagle method par se, but is a product which derives itself from several 
different ^ayt to collect, analyze, and use evaluation information. Chepter III introduces the case 
study method, its advantages and disadvantagp, and its potential uses. 

■ ' ' t . ' * ' ' 
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CHAPTER III 
THE CASE FOR THE CASE STUDY 



Ideas about the case study method are changing. While the case study has been used extensively 
in the medical, legal, and journalistic fields, the social sciences (except anthropology and branches 
of sociology) have relegated it to a lower status than other research designs primarily because of the 
lAck of control groups and perceived limits in general izability. Now, however, the case study it being 
viewed as a way to understand the complexity of a program and its parts. It.deals with information 
about a pjirticular program in a complex, holistic way that reflects the life of that program. "A case ' 
study that portrays an educational problem in all its personal and social complexity is a precious 
discovery."^ Chag^ter 1 1 1 is designed to introduce you to the case study method. 

« «■ * 

The pase Study Defined ^ 

A review of , the liteirature reveals many definition^ of the case study. Some are highlighted in 
Figure 2. Several themes run across the definitions that comprise the attributes of the case study 
method. The themes include: , . 

• Th* Case Study is Particularistic ' 

• The Case Study is Holistic » 

• The Case Study is Longitudinal, and 

• TheCaseStpdy is Usually Qualitative. ^ 

Each of thesd four themei is elaborated belovy. t 

• • Th» CBt9 Study It Ptrtfcultr/ttlc. Case studies focus on events ih one particular setting. As 
itiike not6s, the principle difference between case studies and other methodologies is not^ne of 
method since case studies can use systematic observation techniques (e.g., 1ntervie\^ing) or*be highly 
abstract and statistical. ~ - 

, , ' the principal difference is one of focus. The case study focuses on the happeinings 
of a single^actor^ a single clattroom» or a single enterprise-usu.ally iinder natural 

^ conditlons-^to ai to understand It, that bounded system, in its own habitat. What 
is b0lng studied Is th§ 0999. The case Is lomething deemed worthy of cipse watch^ 
it his characte^ it has a totality, it has boundaries. It is not something we want :to 
represent by a score. It is not something we want to represent onlyby an erf'ay of 

• scores. It is a complex, dynamic system. We want to understand its own complexity.^*' 



•:Qim $W^y ft- Holtstlc. Th#qaM Itudy tries to capture the totality of the phenomenon. 
In doing ib, the biM s^^ usually portrays the iqterplay of the different factors that are brought to 
bear orv the progrem and tries to pretent the views of the different groups involved. The intended 
result Is 8 rich description and undfrstandlng of the program, its complexity, and its dynamic nature. 



• The Case Study is Longitudinal. Case studies-usually toll a story over Xwwa. This attribute 
the case study finds its traditions in othiunjraphic studios whore the rosonrchor or ovaluaTor spends 
n (irnnt deal of lime studyinq a cilltiire hy hecoiniiv) as much a part of it as possible (e.g., Margaret 
^Vlead in Samoa). However, at times, the case study is a "slice of lite." 1 hat 15. given the needs of the 
users and resource constraints, the case study may not be longitudinal, but TTToments in time. 

• The Case Study is Usually Qualitative. The case study uses a variety of methods -both quanti 
tative and qualitative-to collect information. "In fact, one of the.problems involved in the use of 
the case approach is the breadth of the specific tools that may fie net^essary in the gathering of data 
and the need for the investigator to control them."'^' However, qualitative mgthods (e.g., personal 
documents, life histories, interviews, statistical records) are well suited to tlie case study approach 
because "case studies use prose and literary technique to describe, elicit" images, and^analyze situations 
. . . They present documentation of events, quotes, samples of artifacts, and so on." 



Figure 2 
Case Study Defined 



Case study is a process of research which tries to 
describe and analyze some entity in qualitBtivo, 
coiViplex, and comprehpnsive terms not infre 
quentiy as it unfolds ovei a peiiod of time. 

Wilson 



1. 

Case study is not a specific technique. Rathet it is 
a methocfof organizing data for the purpose Of 
analyzing the life of A social unit. 

Franklin and Osbor"" '^^ 



Case Study is a study of a bounded system, 
^emphasizing the unity and wholeness of that 
system, but confining the attention to those 
aspects.that are relevant to the research problem 
at that time. 



Stake 
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Case study is an intensive, detaHed 3nalysis and 
description of a single organism, institution, or 
phenomenon in the context of its environment^ 

25 ^ 

Anderson et al, 
, — — » ^ 
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In summary, the case study method tries to describe and analyze some pi^ram in compre- 
hensive terms with its idiosyncrocies and in its complexities not infrequently as it unfolds over time. 
The methods used to collect the information will vary based upon th^informatiQn needs of decision 
makers. 

Usafulnms of the Case Study 

The growing acceptance of the case study method recognizes (1 ) the limitations of traditional 
experimental designs and the disenchantment with the all-too-frequent finding, "no significant 
differences"; and {2\ the need for alternative methods to answer questions about certain types of 
educational problems. Thus, importance is now being placed on understanding a program in all of its 
complexity, and the case study method is being perceived as a way to undertake the task. ^ 

For exampfe, McCaslin identified fourteen strengths of a naturalistic approach to evaluating, 
vocational education programs. He posits that naturalistic inquiry: 

e is a legitimate evaluation methodology in that it is structured, disciplined, systematic, 
comprehensive, and replicable; , 

extends the repertoire 6i techniques available to the evalMAt^; # 

• Recognizes the multiplicity of causes associated with given outcom^^in vocational 
education and recognrzes that causes and Outcomes Can interact in a variety of ways; 

• recognizes the range of diversity both within apd b^w^en program results in individuali- 
^ ^ zation of evaluation designs; . 

• allows the evaluation to consider a broader view of vocational education ogtcorpes; 

• ^s a continuousvactivity under^keh through(Hit the process of vocational education; 

• evaluates vocational education program's as they occur; 

• is not constrained to examining olply those outcf^mes amenable to quantification; 

• allows the evaluator to collect tfifotmation on outcomes not known to be important or 
anticipated cluring th'e design qf the'tstudy; 

\ 

• is an evaluation methodology likely fp bcrunderstoodby most people* 
' • provides personal and immediate feedback; ^ ' , 

• can be individualized to meet the needs of diverse people; 

• diminishes reliance on vocational education students to submit needed information; and 

• pi-oduces results which may be more useful to decision-makers. 

In sum, th^ case study, which-usually relies on qualitative methods, provides a richness of ' 
i information that reflects the dynamics of the program. From a methodological vievi^point, the case 
study method should allow evaluatort to avoid meaningless or misleading questions because they can 
0 ) formulate problems at the evaluation progresses, i.e.,-they arfr'not bound to a priori questions; 
(2) modjfy data catagories constantly, making them more suitable to analysis; and (3) reduce the 
distortion of variables that may be difficult to quantify ^y using quantitative methods. Last, the case 
study . is perceived as having heuristic value. The information obtained through a case study is both 
rich «n^ rewarding and may be used to define concepts, generate hypotheses, and ground new theory 
for further testing. 

1 ^ ■ * ■^j 
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Lltrnltatlont of the Case Study 

While niuch of the usefulness of the case study rests upon its intuitive appeal (e.g., the readers 
expandvttioir experiences by vicariously sharing those of others; rondors exporience the viewpoints 
of several different actors by having an ^valuator probe, interview, and analyze the program; and readers 
experierice the entire program and all of the factors that have an impact on the program),. there are 
some limitations to the use of the case study which should be kept in mmd as a case study is planned. 

Case studies^may appear to be simple to conduct at first glance, but they are not. It is important, 
for example that the evaluators know how to interview, to sample case study sites scientifically, and 
to design coding systems that allow for the retrieval of the information. The case study method, while 
designed to be expansionist rather than reductionist, can expand to include alrtiost everything and 
rather quickly at that. Thus, the case must be defined and given boundaries. In all evaluation efforts, 
different segments of the audience will request and relate to different pieces of Information. As 
Wilson «ated: ^ \ 

Usefulness is related to the role of the readep. By it5 very nature, a case study ^ 
provides information on the actions and perspectives of members of a variety of 
groups However, given the limitations of length, Writers of case studies have to 
decide on whom to concentrate. They can opt for equal treatment of different 
. groups or emphasize! Certain perspective^; We [Center for New Schools] have 
discovered that the logic of waiting a case study may not match the logic of 
usefulness to the reader. Any given reader will want more detail about perspec- 
tives and actions of particular kinds of actors (usually people in his/her own role) 
that would make sense in the overalUase study. An inevitable lack of detail arises 
from the difficulty of having a balanced case study focus sufficiently on the 
reader's role.^° 

Associated with the difficulty in matchihg information to readers is the problem of the length 
of a case study In order to present the complexity of a program, and to chronicle the concrete daily 
«S over time, case study repprts tend to be rather lengthy; ahd, since they require a long time 
to ead they are often given low priority among the other demands on the reader s time. The third 
L fourproblern^^^^^^^^ aLe may be alleviated by making certain that the intended audience 
r^ei^Z case study report. Condensed versions of the case study that extract pertinent portions of 
the report for segments of the audience may be helpful. 

" Case studies tend to be most interesting to "people who already value some of the approaches 
. illustrated. Additionally; if the approaches are valued, case studies are more likely to lead to action. 

The last limitation that is found irt the literature is generalizStjility. The question is often raised: " 
How can information from 6ne case study setting be useful in othep settings? To rephra^ the 
auwtlon. how can we generalize from one case study to several cases? The case study has no handy 
auS lve measures of generalizability. The issue of generalizability In the end is related to what 
L u«? t^^^^^ tofearn f rom the case study. The differences ir^ sites do not 
case study in^elwant In all but the site studied. Rather, the case study should provide enough detail 
aSd de^rlption about the uniqueness of the case study site so that the user can determine the 
differences and similarities between the case study site and his/her own site. 
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Tht Cm§ Study and Vocational Replication 

The use of the case study in vocational education is just {beginning. For example, in 
1978 the National Center for Research in Vocational Education conducted a rtiuUi site case study 
to identify and detcribe tbe factors associated with the differences in job placement rates as reported 
by states,^^ A second case stu<^y was undertaken in 1979 at eight sites to identify and describe the 
factors that affect placement rattfs in secondary vpcatjonal education programs.^^ During 1980, the 
case study will be expanded by focusing on the postsecondary level. 

Case studies also have been published in the education field and in the human resources area 
dealing with the world of work. For example, the Northwest Regional Education Laboratory has <^ 
%used the caSd study rnethod as part of its evaluation of the Experience- Based Career Education Program; 
^oiiis Smith and Paul Pohland used a naturalistic approach to study computer-assisted instruction 
programs; the Centef foe New Schools used descriptive case studies to evaluate a Documentation and 
Technical Assistance Project; the Education Commission of the States conducted case studies in 
fourteen sites focusing on strategies implemented to improve rural education; and the National Institute 
of Education is currently funding case studies in four sites dealing with rq^al vocational education 
programs. Studies of Comprehensive Employment and Training Act (CETA) programs also have used 
ihe case study method. 

At this point the reader should feel comfortable with understanding the case study method, 
its advantages and limitations, and how it fits with other evaluation iK^tivities. Chapter IV focu^ on 
different techniques that may be'used to collect case study information. 
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CHAPTER IV 



COLLECTING INFORMATION FQp 
CONSTRUCTING A CAS&.STUDY 



These are many techniques to collect information, sucli as questionnaires, surveys, interviews, 
■^and observations. The techniques evaluators choose really depends on the fit between what questions 
they are seeking to answer and which teclmiques best meet the data-gathering needi in^plied in the 
question. Of course, constraints stemming from the available resource level that may be applied to 
the evaluation and the political environment ihust be considered as designs are made. This chapter 
provides the reader with a brief discussion on evaluation methodology and then looks jat three 



Two theoretical perspectives have dominated social science methodology: \og\ia\ positivism and 
phenomenology. The logical positivists, such as Auguste Comte and Emile Durkheii^, seek "the facts 
or caus&s of social phenomena with little regard for the subjective states of individuals. Durkheim 
advised the social scientist to consider 'social facts,' or social phenomena as 'thingsl that exercise 
an external andcoercive influence on human behavior."^® The phenomenolodstsjsuch as Max Weber 
and Irvin Oeutscher, oil the other ^nd, are concerned with understanding h^m^ pehavior from 
edch person's point of view. Thus, ^he phenbmenologist views human behavior asNi product of how 
a person sees the world. Phenomenc^logists believe that human behavior cannot be understood without 
underetanding the framework within which those being studied interpret their actions, feelings, thoughts, 
and nDotives. \ 



Tha differences between logical positivism and phenomenology are important. Social science in 
the United States strove to follow the logical positivism stance of objectivity and deductive reasoning 
by attempting to minimize subjectivity and standardize the interpretations attributed to the data. 
Phenonienologists, on the other hand, see a number of dj^fferent structures that may be designed to 
understand the data they pollect. In fact, they espouse the belief that perhaps the most important 
structures to understand might belong to those being studied, rather than those constructed by 
evaluators «pr/{>r/l ! 

• ' ; . 

} T 

The lbgk;al laosltlvists and the phenomenQlogists, then, approach different problems ^nd seek 
different answers^ .T|)e research methods they use are oftervdifferent. For example, logical 
posi^iviitt search for facts and causes llfough demographic analysis and surveys which produce * 
quantitative data that is.statist|(ially analyzed to prove or disprove relationships among variables. 
Phenomenoiogists, however, often use qualitative methods (e.g., interviews, observations, unobtrusive 
dji^i which reiult \n descriptive data that display how those being studied view their world. 




A Few Words About Methodology 




What it tht CaM Study i^ethod? 

At this point, it might be assumed that the case study method solely uses qualitative data. This 
is only partially tiue as Morris Zelditch stated: 

There it, In fact 'a tendency to be either for or agninst quantification, as if it 
were an either/or iMue. To some extent the battle lines correlate with a relative 
concern for 'hardne»f' versus 'depth and reality' of data. Quantitative data are 
often thought of as 'hard' and qualitative as 'real and deep'; thus if you prefer 
'hard' data you are for quantification, ^nd if you prefer 'real, deep' dfita you 
are for qualitative participation observation. What to do if you prefer data that 
are real, deep, and hard Is not rmmediatelyjpparent.^^ 

' McCaslln also cautions against those individuals stating the need for 'hard' data and discrediting 
'soft' data. He warns that:' ' \ 

These people often fail to realize that there Is bad 'hard' data as well as good 
' 'soft' da^a. Measurement through ngmbers alone is not the only way to extend 
or to solidify oar understanding of vocational education. 

Zelditch continues: \' » 

A more fruitful approach to the| issue must certainlv recognize that a field 
study Is not a single kind of information. This approach suggests several crucial 
questions: What kinds of methods and what kinds of Information are relevant? 
How canvthe "goodness" of different methods for different purposes be 
evaluatec4?3% .# \ ^ ^ 

Thus the case study, while leaning tovfard the phenomenology tradition becaus^^lt well 
suited to using qualitative methods, does n<it rely solely on the methods'used by phenomei^o ogists. 
Qualitative methods can complement the iriformatlon collected from other methods by conf lrmir|g 
findings and locating discrepancies. This caii lead the ^valuators to study, weigh, and analyze the 
findings carefully to understand the discrebancies. An evaluation of an.£xpenence-Based Career^ 
Education Program (EBCE) by the NorthXest Regional Educational Laboratory tried the case study 
approach as a portion of Its evaluation effjart. It reported that: . ^ 

^ a casestudy Is a useful complement to a totdlpacltage of methodologies 
in the evaluation of a program Ilka EBCE. It provides an empirj^escription 
of the 'treatment' in a traditior'ial pretest-treatment-post-test (^n. This 
documented description of what the program 'is like' is useful in interpreting 
e}5per|mental findings as well as giving an evaluation audience the vicarious 
experiences of 'being there'.*'* ^-^.^^ 

While there are several publlshed/texts that focus on quantitative methods,*^ the number of 
texts written on the use of qualitative methods is more limited.*^ Therefore, the remainder df ^hla 
chapter It devoted tq Introducing three t^ualltative methods: observing, rhterviewing, and gathering 
data unobtrusively. 



Qu/ilitative Methods ' 

As Bogdan and Taylor suggest, "qiialitative methodologies refer to research procedures which 
produce descriptive data.'"'^ When using qualitative methods, settings are described as a whole. ^ 
There are three techniques that are well suited to gathering this type of information: observing, 
Interviev^ing, and gathering data unobtrusively. Each technique is introduced below (see Chapter VI 
for a discussion of how to apply each technique). 

• Observing - Observing is something we do naturally, and as such it undergirds the two 
other techniques that will be discussed. By definition, observing refers to that research ' 
"characterized by a period of intense social interaction between the researcher and the 
subjects, in the milieu of the latter. During this period, data are unobtrusively ind system- 
atically collected. ^ 

Observing is based upon three activities: looking, listening, and asking. These three 
activities when interwoven form the Complex process of observing: 

Events and happenings connecting one activity stimulate new action 
in the other two activities. Things seen and heard stimulate questions 
to ask as well as indicate new things to look and listeh for. Questions 
asked arid answered stimulate new lines of looking and listening, as 
^ V well as new'questions to ask."*^ 

r • Interviewing — Interviewing is more than merely asking questiohs of people. Interviews 
may vary ftom informal moments when you have the opportunity to talk with people 
involved in the project to a carefully nurtured relationship with a person knowledgeable 
^^bout the situation in question. Specifically, interviews may be divided into two types: 
unstructured and structured. Unstructured, or elite,"*^ Interviewing is probably the most 
artful way of interviewing. Briefly: 

It is a purposeful discussion between you and another person, but ■ \ 
Without specific prestructured questions. The emphasis is on listening 
as you discuss things that are important to the interviewee [within 
^ the context of the program} . The intent of the unstryctured interview 
is to elicit from the person being interviewed a highly personal and 
detailed account of particular events or sentiments related to the prograni.^^ 

' The structured interview consists of a set of pre-determined questions, like a questionnaire, 
that are verbally asked of a person. Some questions might anticipate responses that fall 

. into categories (e.g., vocational education service areas of training: trade and industrial, 
distributive education, home economics; or they may be open-ended, such as "Why did 
you enter t||is vocational education program?''). 

• Gathering Data Unobtrusively — When data are collected unobtrusively, the evaluator 
does not necessarily have any direct contact with individuals such as when observing or 
interviewing. In this case, the information may be gleaned from: ' 

• physical traces, or observable signs of the use of objects or space in a setting (e.g., worn 
carpet around a particular display); 

• artifacts, or objectives that can be identified as resulting from human activity (e.g., 
lesson plans); 

• records (e.g., minutes of meetings, correspondence, grade books); 

• indices, which are comparisons of statistical data (e.g., ratio of males to females in 
selected classrooms); and 

• body language. ' 
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The first four chaptert of this handbook are designed to/rovlde the reader with a framework 
for understanding the case study approach. Part One discussed (1 ) the purpose of evaluation and 
some of its problems, (2) the natural istic .approach to evaluation, (3) the attributes of the case study 
method, and (4) techniques to collect c4se study information. 

We are now ready to move Into Part two of the handbook-how to conduct a case study. 
However, before you begin tp read Part Two, take a few momerits to think about the information 
contain^! In Part One since the case study approach may be a new way for you to look at evaluation. 
Therefore, give some thought to- how you might use the case study method in your work. Worksheet 1 
should help you do that by focusing on your evaluation responsibilities using the journalistic questions 
Whd?'What7 When? Where? and Why? Then, on to Part Two to figure out How! 



WHO 
(is the audience?) 


Conci 

WHAT 

(information 
will l3e CQllected?) 


Worksheet 1 

iptuallzing a Case S 

WHEN 

(will the case 
study be done?) 


tudy 

WHERE 

(what sites wilL 
be studied?) 

V 


K 

t 

WHY, 
(do a case study?) 


1 

\ 

\ 

W ' 


I* 
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CHAPTER V 

THE PRE FIELDWORK STAGE . ' 



In any avaluation, several steps need to be taken before the first piece of information is 
collected JormaHy. Thll, Qf coune, hoJdt true for a case study. Thq preliminary steps in designing 
a caia study art: Mttlng boundariat, defining the unit of analysis, selecting a site(s), establishing 
Initial contacts, developing data 6bllectlon systems, and defining fleldwork procedures. Each ^tep 
is ditcuttad Ic^mora depth below. 



StapOnf: Satting Boundaries 

After the broad question of the case study Is identified, the evaluator faces what Guba^^ calls 
"the boundary problem" or, in Pattun's terms, "cultural trade-offs".'*' This means that the d^irablllty 
of studying one or a few questions In great'depth or many questions In less depth must be determined. 
Iji other words, what will be the limits of the case study? How will the limits be selected? For example, 
ihotild all^ parti of a vocational adulation program In the school be studied or only certain programs? 
, Should all students and/or teachers be studied, or should a specific group be studied? Should all 
outcomes be studied or certain specific ones? 

Boundariat may ba sat in several ways based upon tha information needs of daciiion-makers. ' 
They should not ba set by the evaluator alone. For the infomriatlon obtained from tha case study to 
be utilized, it must answer the questions of the parsons who make dacilioni. Therefore, the latter 
mutt play a critical role in identifying the trade-offs and defining the boundaries. . ,^ 

i Boundaries may ba set by geographical arels (e.g., urban/mral, north/south), themes (e.g., types 
of vbcitional education instruction, outooma of a vocational tducation program), or theoretical 
and/or tubitantiva interasti identified at tha case study it f irtf concatvad. All of tha evaluation 
quattidht Idantlflad will probably be important quettioni. Trade-offi will need to be made between 
tha breadth and depth of the cata ttudy given the exitting financial, tima, and other constraints. Once 
the boundariat fpr the case stddy have ba«n established, the second step->determining the ur^jvof 
analytls-^beglns. ^ 

. '-i ^ A f*w words of caution jra In order at thii point. If the foundatlont of tha cate ttudy are not 
adiiuataly laid U.a., Involving dacltlon-makart in defining boundarlM, defining appropriate and 
Ife . pliSlWa boundariat), tha ci»a study will IlkWy fall in Itt purpose ragardlaj» 

procaduni So spend the necettary time now to lay a solid foundation. Worksheet 1 should be 
helpful. 



WbrkthMtl 
Setting Boundaries 

H«v« you: ^ 

1. id«ntifi«d the broad evaluation xtuestion?- 

2. established the boundaries? 

3. defined boundaries that are appropriate? 

4. oonsldered the breadth and depth trade-oft In d^flnlnp boundaries? 

5. involved decision-makers in defining the boundaries? 

6. considered informatidn needs of decision-makers in defining boundaries? 



Yes 


No 


o 


a 


o . 


.. o 




o 


o . 


0 


o 


o 


cr 


0 



SW^: Determining the Unit of Analysis 

Onct a focus of the case study hat been determined, it is time to decide on the unit of analysis 
to be studied. Simply/ the unit of analysis is the "thing" that is being studied, such as individual 
students, types of vocational Mucatlon programs, school systems, or. state education departments. 
The unit of analysis chosen depends upon, what Information is desired from the case study. For 
example,. If you are interested in why placement rates vary, the unit of analysis might be stu^ 
or the specif ic program areas (e.f., distributive education, trade and industrial) might be the 
analysis if you want to understihd differences in placement across program araas. School syi 
might be the wit of ahalysis It you want to understand factors affecting placement in different 
envlronmentsTwhatever unit of analysis is chosen lends itself to specific kinds of data coj lection an 
analysis techniques that best suit answering the evaluation questions. 



Worksheet 2 is designed to help you decide whit unit of analysis is appropriate for a case 



study. 



r 
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Worksheet ;2 
Determining the Unit of Analytit 



1. Description of Isoundary: 



* * 

2. Evaluation question to be answered: 



3. Possible units of analysis: 

(1) 

(2) 

(3) 



4. The most optimal unit of analysis: 



6. Why was it chosen as the unit of analysis? 



'^6. Will the unit of analysis se^jected be abie^to give the needed information at the end of the 



case study? Why? 



" ,;• ■ • V 1) .. ■ 29 



33 



$tepThrM: Selecting a Site(t) 

After decisions have been made regarding the questions to be answered by the case study and 
the unit of analysis, it is time to decide on what sampling methpd to use. Briefly, snmples may be 
drawn that are.r«/»c/o^7j/i.e., all potential members of the sample have an equal opportunity to be ■ 
Xted by chance; or purposeful, i.e.. sites are selected because theV contain certain characteristics. 

Random samples generally are used when you want to generalize the findings from the samples 
to the entire population. Randpm sampling increases the likelihood ^hat the information collected 
In the study is representative of the entire population. Three types of randbm samples are often used . 

^» Simple random sample - all members of the population have an equal chance of being 
selected. 

• Cluster sample - the selection of sarnple members are chosen from the total population 
In groups or clusters rather than as individual sites. The clusters selected are often 
natural or administrative groups of the population (e.g., automotive programs, health 
programs). 

• Stratified sample - the population is divided into subpopulations or strata and a sample 
is^selected from each strata (e.g.. the automotive programs in small; medium, and large . 
regional vocational high schools). 

Jo summarize, random sampling procedures help to av6id bias in the samfile because sites are 
selected by chance. Simple random samples should allow theevaluator to obtain a relatively repre- 
wntative sample that permits generalizations to the entire population. By using a cluster or stratified 
sample, the evaluator should have increased confidence in making generalizations to particular sub- 
groups or areas. . ' 

Purposeful sampling is used when the evaluator wants to learn something, or understand somp- , 
thing about certain select sites, and does not need to generalize the findings to an entire population. 
To do this, certain informatiqamust be available about the site so that the evaluator is aware of the 
variations among the sites. For example, Marvin Alkin et al. undertook a case study on the utilization 
pf evaluation information in five schools. They stated: 

, In locating potential sites, the researchers capitalired on a variety of resources. 
Personal and professional contacts in public educational agencies were especially 
helpful in directing our attention to potential sites. The approachabiljty of the 
site personnel was a clear consideration since we did not h^ve the power or the 
inclinations to impose ourselves on the case study hosts. As a resuU, we cannot 
claim that the five chosen sites were randomly selected. Nevertheless, we sought 
to avoid 'showcase' programs and were able to balance out our sample with a 
variety of program types . . .'^ 

'Several different strategies may be pursued when drawing a pMrposefu I sample. F'rst, 
extrerm cases may be sampled when the most information might be gained by studying poor Programs 
. a^d^eptional ones. The case itudy then attempts to understand the situation under which programs 
' SeveloMail, or succeed. In gampling extreme cases, the evaluator needs to identify those sites where 
the most can be learned; thpse sites are selected for the sample. 
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A second type of purpoieful tampling is the typical sample. In 'tHle Instance, information is 
gathered about typical programs, and the findings cannot be dismissed beei^se the program is known 
- to be special in some way. However, the findings cannot be generalized in the tr&(!frao'nal sense of the 
» term. One word of caution should be interjected hero - the evaluator must work with decte.ion makers 

to reach consensus on the attributes of a typical site. 

k " ( 

Another sampling strategy Is to maximize Xhe.variation among the sites. For example, samples 
are often drawn to ensure that all geographical sites (e.g., urban and rural) or demographic features 
(e.g., site size, type of vofcatlonal education programs offered at the site) are represented in the temple. 

By attempting to increase the diversity or variation in the sample^ the evaluator 
will have more confidence in those patterns that emerge as common sites while 
at the same time biing able to describe some of the variation that has emerged 
to make prograips unique as they ^dapt to different settings.*^ . 

Fourth, the sample might consist of critical cases, i.e., those sites whtth can make a point 
dramatically. For example^ many new commercial products are test-marketed (n Columbus, Ohio, 
on the assumption that If the product does not sell in Columbus, it will not sell nationwide. The use 
of the critical case is especially important when resourcesjimit the number of sites under study. By 
sampling criti(fal cases, the findlniEis may be logically general1;ced to other sites since the critical case 
Is selected on the assumption that if the. finding js true in this case, it is jikely to be true In all gases. 

Fifth, you may decide to sample politically important or sarjsitive cases which might draw 
/^attention to the^ase study or avoid drawing attention to the case study by intentionally eliminating 
politically sensitive sites from the sample. This is a variation of the critical case strategy, especially 
given the fact that th? critical factor might be politically sensitive. 

■f-,- > 

Last, samples may be drawn by convaniat^ce, I.e., those sites which may be studied most easily, 
this type of sampling is usually the most common, yet the least deiirabM. While the constraints 
stemming from costs, time, andotherTesources are real, they should b/considered last, after the best 
strategy is identified to collect the desired information. 

In summary, purposeful sampling Increases the usefulness of the findings obtained in small 
samples because sltef are intentionally selected^ for c^taln traits. I.e.^ extreme cases, typical, maximum 
variation, critical oases, politically important or sensitive cases, or coAvenienctf. 

Worksheet 3 will help you choose the type of samjiiing technique to be used in the case study. 

\ 

\ 
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WorkihMt 3 


, (Li 






> < 


SftlMtino II SitfiU) 


• 


• 


1 


W^M the sample b« drawn: 


\ 




Yds 




• randomly? 


r 




o 


o 


• purpoMfully? 


At 




o 




If a random sample it telected/wlll It be a: 








• timple random Mmple? 


• • 




r\ 

\J 


o 


• clutter tample? 






O 


o 


• ftratif led sample? 






o 


o 


, • extreme case? 




* 




o 


• typical case? 






o 


o 


• maximum variation? 






o 


o 


• critical tase? 










• politically infH>ortant or tentitive cate?. 




o 


o 


' • convenience? 


• • 
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Step Fourt EittbHthing Initial Contacts 

Once the sites hav« been selected, it is time to gain their approval for the case study by 
establishing initial contacts within the site. Many evaluators find this part of the case study to be 
unnerving, but it does not need to be. A few hints should help. First, when dealing yvith the 
''gatekMptrt'- (those panont who have the power to grant Of deny access to the site) and other 
pertont at the site, always tell the truth. Second, carefully explain the purpose of the case study and 
emphasize that you are interested not in the particular site bjut rather in sites in general whlth are 
similar to the one you are asking permission to study. Third, follow the protocol network. That is, 
be careful t^at the "right" persons at the site are contacted initially by looking at from the top down. 
For example, permission would be needed from an area vof:at|pnal school director before one of the 
voc^t^onal programs in the school may be studied. Lest, remember that the gatekeepers and others ' 
have not been as involved in the case study as you. Thus, a general easy- to understand explanation is 
more suitable than a complex, long-winded, specific explanation of the case study purposes. As 
Bogdan notes, "the rule then Jn trying to get in Js to be honest, but vague or imprecise. Present your 
purposes in a way Which will not harm or produce anxiety in. your subjects." 

At this point a conr^ment about confidentiality is appropriate.^Before the gatekeepers will agree 
to participate they will want to know what the evaluator plans to do with information collected . 
and who might have access to it. ' 

t, - • 

Gatekeepers and other subjects often want assurances of confidentiality in regard 

to the name and location of their orgaTiization, as well as themselves, In any 
. reports that are written. They also want assurance that the information you k 
gather will not be used in any way to harm them. The code that you are going ■ 
to live by in regard to this should be made clear to them from the start.*^^ 

Confidentiality may be difficult to ensure in tome instances, such as when only one,site is under 
study or the number of participants i!i few. ^ 

Where confidentiality cannot be protected it i^ln»ortant the the evaluator^make 
it clear to people being -interviewed and particir^amsbQing observed that, v^ile ' 
people will not be Identified specifically in a repoj^ It rV)ay be possiblii^t9 identify 
them from the daicriptiort Qf what thay have done or qiiotationi of what they , . ^ 
have said." 

Regardless, the way in which confidentiality will be handled must be nagdtiated before any dfata Is^ ' 
collectfd: ► ^ 

<• * , ■ 

FLgure 3 provides an example of 'asfablishing initial^contacts from an actual case study. fi^Hvr 
you hwTa read this scenario, design yb^ir plan for establishing initfi^,contacts using Worksheet 4. 
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Figurt 3 

Establiihinfl Initial Contacts - A Scanarlo 



Initial contact with a potential host was »ftabli«h9d by the lenior Researcher either with the 
site program evaluator, program director, or ichool district administrator, and arrangements 
were made for an exploratory meeting beritween the researchers and program personnel. This 
meeting served mainly to introduce the researchers to the progrQrti administrators, to lillow the 
potential hosts to look us over and assess our intentions. Little attempt was made to actively 
question the program representatives present; instead, our concern was with answering their 
iguestions about pur research intentions. We tried to project a sincere and nonjudgmental 
interest In their program and in the way that prograrp evaluation had taken piaqe In their 
school. We ma(le the point that thty would benefit from a better understanding of the way 
that evaluation, as a process, had functioned in serving their needs-and wv would benefit in 
a fuHer understanding of the evaluation process as it occurs at a variety of school sites. ^ 

Anonymity was promised the participants. In the various reports emanating from the 
study including of course this book, fictitioul names have been substituted for the actual names 
of citlii, districts, schools, programs, and individuals, and other potentially identifying elements 
have bMn disguised in ways which help presen/e the anonymity of the partteipants Without 
distorting the relevant facts. Individual actors in each case situation can, of course, recognize 
their coparticipants' identity, so that this mechanism does not shield a parttolpant from his or 
her colleagues. As an ethical matter, we did seek to safeguard confidences shared with us. (The 
tension Tsetween fidelity to the situation and personal trust and rapport it real and problematic, 
but in these studies we were able to honor confidences without sacrificing the integrity of the 
research effort it^lf.) 

Once th,e program representative came to feel that our interwt in what they had to tell 
us was sincere, and once they became convinced that promises of anonymity were not just a 
game we were playing, they began to talk about their programs. We did not try to force these 
initial remarks other than to indicate our interest in learning as much as possible about the 
program and its history arfd context as well as the story of the program's evaluation. We did not 
tap«*record these (or any subsequent) meetings, although we occasionally jot'down notes if 
' that seemed appropriate and nonthreatenlng. (These on-the-spot notes, were usually elaborated 
upon and -expanded after the field worker had left the interview location.) 
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Workthi6t4 
Eitabliihing Initial Contacts 



1. Who araHhe gatekeepers? 



2. What it the protocol network?. 



T- 



"^3. What are the options for making the Initial contact with the gatekeeper(s)? 



4. How will the case study be explained? 



5. Will the initial meeting with the gatekeeper(8) be designed to defuse their concerns? How? 



6. How will confidentiality be assured? 



Sttp Five: D«v9loping Data Collection ProcadurM 

Chapter III introduced tliree ways of collecting quillitalivo information Uirough systematic 
observation! observing, Interviewing, and gathering data unobtrusively. By observing, searching aixi 
discovering clues, interviewing, and interpreting evidence, a meaningful picture about the program 
' or periont under itudy should emerge. That Is, answers will begin to appear to questions such as 
thete: What is heppening? How are things happening? Under what conditions are they happeriing? 
Why are they h^penlng? ' . 

» • . • ' ^ • • ' 

Which iystematic observation methods are chosen will depend upon the purpose of the study 
and the decisions made in the prior steps. For exaijiple, the unit of analysis initially will help d^ide 
what to observe and whom to interview. Through observing an>J Interviewing, other situations or 
persons will be identified for further observing and interviewing. ' 

Qene^ally, all three lystjwnatic observation methods are used in a case study. Each method 
produces a different way of looking at the same phenomenon, ^hich will confirm or cor^tradict 
other findings (triengulatlon) and will lead the case study team to new areas of inqtlfry. Briefly, 
triangulation "is simply the process of obtaining Information from several different sources (e.g., 
student and supervisor) or from the^ame source (e.g., student) using different methods."'*^ 

There are several publlcatlpns available that describe these different techniques at great lengt|».' 
Worksheet 5 can be used to help think through the data collection procedures. 
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^ Worksheet 5 

Developing Data Collection Procedures 

\ 

1 yjhlrh prnrnHiirAit ar« appropriate? Why? 


% 

\ 






' ' • Intervlewlna? 

• Observlna? 

• Gathflring data unobtrusively? 

9 Who will hA Interviewed? ^ 

^ What u/ill he ohsarved? 

d What unnhtniii\/o data will be collected? 


t 






K hAuIi Willi the pro<iedurB!i be triangulated? 

i ■ — 


i 
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step Six: Orgtniiino Data " 

Just at quantitative data are collected with predesigned codes and put into the computer for 
analysis, the data from a case study must be collected and stored in a format that makes it easily 
retrievable. The coding system should be one in which (1) the data ate easily available for analysis,, 
(2) is the least time-consuming, (3) is easy^to impjiment, and (4) is cost effective. 

Sinca the coding system must be specificaUy designed for each case study in order to addr^ 
the specific evaluation questions, unit of analyds, and data collecti6n methods, it is hot possible to 
borrow an existing coding system. However, "J/alking through" a sample coding system will reveal 
the logic behind how and why a coding system |s desigtied and used. 

Before a eample coding system is designed we will need to conc^^ and design a fcase study. For 
the purpose of Step Six, we are in the evaluation office of the Oneanda education department. Oneanda 
is concerned with the placement rates in its secondary level vocational education programs. While 
the statt department has conducted several evaluation^tudies dealing with placement and has continued 
to collect follow>-up information that is accurate and detailed, they do not know exactly what factors 
affect placement rates. Thus, the evaluation office has been asked to answer this broad question using 
a different approach in hope that they will be able to obtain an understanding of the placement 
process. Through discussions, it appears that a naturalistic approach would enable the evaluators to 
look at different schools in the state and thereby identify different factors that mi^t affect placement. 

Interyiewing was selected a|the primary data collection device because the evaluators felt that 
each interviewee slpuld be able to discuss the placement process freely without pre-structures questions. 
The interviews will be complemented by both observations and data collected unobtrusively (e.g., 
dok:uments). The observations and unobtrusive data will be used to identify questions that should be 
asked and to stimulate the evaluators to new lines of inquiry. Also they will be used to confirm or 
refute information obtained from interviews. 

The unit of analysis of the case study was the school. To get the most useful information 
possible a purposeful sampling technique was selected. A variation sample was chosen because the 
evaluators believed there were demographic considerations that might highlight the factors that 
affect placement. Thu8, the sites included in the sample were sheeted on an urban/rural continuum 
and low piiscement/high placement continuum. The sampling fnRe resulted in four cells: 



Urban 
High Placement 


Urban 
liow Placement 


Rural 
High Placement 


Rural 
Low Placement 



After the Sites were selected, the protocol network was identified and initial contacts were made 
with the gatekeepers. 



Now let us design a coding system for a case study ^« that seeks to understand what factors 
affect Dl^ement rates in Oneanda, The first step in designing the coding system is to designate a 
mlmis by Sw^c^^^ of datum will be classified. In this example, there are five broad categor.es. 

[Mp«t a scheme needs to be established to identify the schools participating in the case study, 
the evrLr^oirc°ing the information at the site, the type of person (e.g., teacher, student, adm.r^is. 
;l:,::rrg\m:re Jed or Observed and the data coiiecti 

document review) Figure 5 lustrates these codes. By lookmg at 1-igure o, may ut. 

«?re estabrhed for the school (or site) being studied, the program the ,n,erv,ewer, the typo of 

interviewee, and the data collection procedure used (e.g., mterview). 

Third each interview, observation, or document reviewed must be assigned a sequential number 
, „ ?m»ster hst This record will serve as the permanent log of all interviews and observations 
rrdeTas we II as „^ I Icum^^ reviewed, ihe master list is divided into three parts: intervi^js, 
Z,^a?ions and documents. Each part is then divided into information obtained a the school and 
. WormatTon obtained outside the school (other). A sample master list appears ,n Figure 6. 

?u Mcoc of .t;,tP and local officials or by all counselors and all prmcipals, or all state advisory 
tiers appear in Fijgure 7. 
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Figur«4 

Sample Method To Clauify Data #1 



Background 
B1 



B2 
B3 
B4 
85 

Definitions 

D1 
D2 



D3 
D4 

D6 ^ 
06 

Placement Process 

PI 
P2 
P3 
P4 
P5 

9 

Function ofP/acement Rates 
Fl 

P3 

Utilization of Placement Rates 
Ul 

U2 , 
U3 

U4 - • ■ 



Delivery System 

1.1 number 

1.2 vocational education 

1.3 structure 
Finances 
Attitude - State 
Attitude - Local 
Cooperative Education 



General Information 
Official Definitions 

2.1 student 

2.2 corfipleter 
- 2.3 leaver 

2.4 dropout 

2.5 relatedness 

2.6 6ode 

Used 

Influences 

Communication 

Monitored 



General Information 

Description 

Policy 

Guidelines 

A8sistar>ce 



General Information 
Philosophy 
Coordination 
Services, Levels 



General Information 
Dissemination 
Com^pliance 
Used for 

4.1 personnel 

4.2 program 



Figura 5' 

Sample Mathod To Clauify Data U 2 



School 

AS 
DV 

Program 

HE 
OE 
Tl 

Evaluator 

E 
F 
S 
R 
M 

Interviewees 

T 
U 
V 

w 

X 



Austin High School 
Davis High School 



Home Economics 
Distributive Education 
Trade and Industry 



E. Weiss 

F. Sanchez 

S. Washington 
R. Bern$tein 
M. Marciano 



Administration 
Support Staff 
Program Staff 
Teachers 
Advisory Council 



Data Collection Methods 
I 

0 
D 



Interview 

Observation 

Documemt 
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For interviews in _ 
School Level: 



AS 



(tohooO 



Other 



For observations In 
" School Level: 



AS 



Uchool) 



Other: 
For, dotiyments in . 

\ 

School Level: 



AS 



(school) 



Other: 



Figure 6 
Sample Master List 



1 

2 
3 
4 

10 

1 

2 
3 

10 



2 

10 % 
1 

10 



1 

2 

10 

1 
2 



Industrial Arts teacher (Snriith) 
Trades and. Industry teacher (Wingston) 
Principal (Johnson) 
Counselor (<3eokowsk I) 



<J|bhamber of Commerce president 
Mayor 

National Industries Inc. personnel officer 



Business and Office class 
School lunchroom 



'I 



Degree of activity in local employment service office 



Local plan 
Follow-Up reports 



City economic plan 
CETA plan 



10 



r 



Figure 7 
Identifying Data 



Example: E. Weiss is interviewing a teacher at Austin High School about the trades and Industry program. 

- W2 



School — — 
Program 

Source 

Interviewee 
Interviewer 
Card No. — 



AS - Tl - 

J 



1 



E 



/ 





Key 




Figure 


School 


' Austin M.S. 


AS 


5 


Program 


Trade & 1 ndus. 


Tl 


5 


Source 


Administration 


W 


5 


Interviewee 


Principal 


,,,2 


6 


Interviewer 


E. Weiss 




6. 


Card No. 




1 


« ' 



* each new card is sequentially 
numbered 



Example: M. Marcianq is reviewing the student orientation booklet fey Austin High School's home economics pr9gram. > 



School 

Program 

Source 

Document Np. 
Reviewer — 
Cerd No. 



AS - HE - T - D23 



M 1 



i t 





Key 


Figure 


School 


Austin H.S. AS 


5 


Program 


Hon^e Economics , E 


5 


Source 


Administration T 


5 


Document N,o. 


Student booklet D23 


6 


Reviewer 


M. Marciano M 


5 


Card No. 


V 






* each new-card is sequentially 
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Figur* 8 






Sample Cover Cards 




S»mph 1 








AS~TI~I~W2-E 1 




[ 


4 

(Interviewer): E.Weiss 
(Interviewee): A. Wingston 

Teacher 

Austin High School 
415 North Street 
Nawton, uiq York U8oo2 
, <413) 894-7382 






<Date): 3/15/80 










OafnfJitf 4L 








AS-HE-T-Dil3-M . , 1 

* 


• • 




{Citation): Davis High School 

Student Orientation Handbook ' 




- * 


(Where/how obtained?) 

UDtaineQ irom porne economics teacner 

(Reviewer): M. Marciano . 

(Date): 6/30/80. ' ' 






* 





Weiss' lecond card will then have a piece of data on it, qpded on the left side of the card according 
to the eodes iri Figure 4 with the indicator riumber at the top. Figure 9 illustrates the second card. 
, Subsequent cirds wHI be coded in the sam» fashion and sequentially numbered. 



t ■ • 



Figure 9 ' 
• Sample Information Card 

•AS~TI-I-W2:-| ^ ; ' 

06 ' Rer queitlon monitoring u4 of tenths, teacher feels the "slow kids" are getting 
, shafted because ibey don't respond as often or with the accuracy of other 
, . students due to confusion of follow-up forms. 



Worksheet 6 is a check list of preliminary itdps to be taken before data collection begins. 



Worksheet 6 
Organizing Data 

Have you: 

1) determined the boundaries of the case study? 

2} determined the unit of analysis? ' ^ 

3) selected the sites in a systematic way? - ^ 

4) ettabliihed initial contacts? 

5) developed a d^ta collection system? 

6) «(^eloped a system to organize data? 
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Yes 


No 


n 


n 


O 




o 


o 


o 


o 




o 


o 


o 



NotM to Chaptsr V 



48. <3uba, Toward a Theory. 

40. Michael Quinn Patton, Qualitative Methods (Beverly Hills: Sage Publications, Inc., 1980). 

50. Although the term sitois) is used here, the same procedure may be used for selecting Individuals 
within a single site. The materials introduced in this step are drawn from Patton, Qualitative 
Methods, Chapter 5. 

/* 

51. Alkin et al., Usir^g Evaluations, p. 36. 
62. Patton, Qualitative Methods. 

53. Robert Bogdan, Participant Observation in Organizational Settings (Syracuse: Syracuse University 
Press, 1972), p. 16. 

EW. Ibid., p. 16. 

55. Patton, Qualitative Methods. 

66. Alkin et al.. Using Evaluations, pp. 36-37. 

57. S. Malak, J.E. Spirer, and 8. P. Land, Assessing Experiential Learning in Career Education 
(Columbus, Qhio: The National Center for Research in Vocational Education, 1979), p. 56. 

58. See note 42 for examples of publications. 

59. The coding system discussed in Step Six is drawn from coding systems used in case studies at 
the National Center for Research in Vocational Education. 



CHAPTER VI 

' THE FIELDWORK STAGE 



* % 

Onot th« evaluation quattions ara dafinad, tha unit of analysii choian, initial contact! attablithad, 
a data collaction lystam daiignad, and fiald work procadurat defined, it is time to begin the second 
stage where the data ara actualtyrottected. The fieidworfc^TTiav'atio tw tnrokeri'down Into a feriei of 
steps: stiff trilntng, ioQlittes of field operations, and data collectibn. Each of these steps is elaborated 
on in the following pages. 



Step Seven: Staff Training 

Evan with careful preparation and complete procedures, the case study evaluators must first 
learn what they are supposed to do. 'The amount and intensity of the training the field staff reoelvii 
depends on the complexity of the tasks, the skills which they potmH^imdi the degree to which they 
will be required to worK independently {^''^"^ 

The training materials should contain step-by-step instructions for what should be done before 
and upon entering tha case study site; sufficient background information to answer questions that 
might be asked about the case study atsthe site; a copy and an explanation of the coding system; and 
clear definitions of the terms and coricepts with which the case study deals. 

• Staff should also be trained in the use of quel itative methods. Not everyone is familiar or 
comfortable using them. When using qualitative methods, the old adage, "practice makes perfect," 

applies.^__.^ 

When, where, and how staff are trained will depend upon the training needs, fiscal resources, 
and time available. For example, a sample training agenda is outlined in Figure 10. Very often, the 
treining may' be combined with, a pilbt test of the case stud^ process'with formal training at the site, 
an opportunity to c6llect data;tind a debriefing. Questions about the coding system and other 
procedures can be answered at the debriefing. ^ 

Regerdless of the staffs experience, it is cruc^l to conduct a training session on the procedures 
to be used in the case stuiiy. It is also a good idea to conduct a training or refresher session on the lise 
of qualKetWe methods. A sample staff training agenda might look like the one in Figure 10. 

The ftrit step in designing training is to identify the needs of the staff. Remi^ber, the best 
•procedures will not be impleniented nor will the desired information be obtained if the staff are not 
adequetely trained. Worksheet 7 is an inventory to help identify and prioritize training staff needs. 
A v^prd Of caution: this it«p is purtial to the successful implementation of the c#e study, so teke> 
the t|m« t<» P(Mrly think put and plan the training needs. Training needs will vary by groups of 
iy|)Uitpmiind^^^ onecMS! itudy to another with the same group of case study eveluatorsi There* 
fort* tailor the training to the spaclfic needs of the case study. Space has been left in Worksheet 7 
to iilbw you to do this. 
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Figurt 10 




Sample Staff Training Agenda 


D»yt 

AM: 


Badcground/purpota of caM study 
naviaw ot now tiiet wora iviacitnj 
Review of acheduie and work atsignmentt 
Discufsion of logittict (e.g., icheduling sites) 


PM: * 


Discussion of coding system 
Sample coding exercise 


Dav 2 

AM: 


I 

' Introduction to systematic observation techniques 
Practice jjiterviewing through role playing 

(tee riflure ii tor exjimpieBj 
Debriefing on Interviewing 


PM: 


Practice reviewing documents 
Debriefing on document review 


Dav3 
AM: 


Practice observing at a sample site 
Debriefing on observing 


PM: 


DItcuttion on the use of systematic observation 
techniques 


Day 4 

AM: 


Each evaluator conducts an interview/observation/ 
document review at the sample site 


PM: 


Codihg morninq effort 


Pays 

AM: 


Review of coding system and case study procedures 
for modification and revlaion 


PM: , 


Review of atsignments 





✓ 

WorkihMt? 






" Idmtifying Mid Prioritizing Training N—dt 






Rank Ordtr* Staff training should Include: 


*♦ 

Yes 


No 


1 ) Background information on the purpose x}f the cas^ study? 




0 


2) >Clear definitions of terms and concepts to be used? 

• 


o 


o 


3) Diicuition of whtt to tlo b#for0 •nt^Hno th^ tlt^? 




-o 


4) Ditcunion of what to do when on tit«7 


0 


o 


5) Procedural to be uted for thft caee study? 


o 


o 


V 

. 6) An explanation of the coding system? 


0 


o 


7) Disousslon/revlaw of systematic observation methods? 


0 


o 


8) Practice with systematic obten/atlon rhethods through 
structured activities (e.g., role playing)? 


0 


o 


— 9) Practice with systematic observation methods in the field? 




o ' 




o 

^ 


o 


1 1 ) Opportunity to modify 

a) case study procedures? 

b) coding system? 

c) persons. to be interviewed? 


o 
o 
o 


o 
o 
o 


12) 








o 


o 


'l3> 






! .. 

1 ^ 


o 


o 


1 

14) 








o 


o 


/ 15) 






« 


o 


o 


^From 1 through 15 with 15 meaning most important and 1 meaning least important 
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Sample Inttrviaw Role Playing 



4 



Code: ER - Interviewer 
EE - Interviewee 



#1. Th§ Super Talker 

Some people control an Tnteractlon by talking tpo long on whatever topics 
appeal to the?n; EE itarti wll(h ER'i flr«t queitlon and gives a relevant answer. But 
before ER can ask ariother dufstlon, EE Is, off on a related toplf, and keeps talking 
just fast enough arid loud ertough to make It awkward for ER t6 Interrupt. 



Some people control an Interaction by not saylnp very much. EE starts with 
ER's first Question and gives a brief, but relevar^nswer. As the intervlevv progresses, 
EE provides monosyllabic response)!. ''^ 

#3. The Missing the Boat Talker 

In this cate, EE does not quite grasp what Is going on, nor why he/she is being 
intervievwed. EE answers all questions and Is trying to be cooperative and helpful. 
He/she Is just traveling in a different orbit, so to speak. 



#4. The Wrong Person 

Every orKe In a while, you find yourself scheduled to talk to the wrong person. 
The problem it that the EE It very proud that Ite/the wat selected. EE responds to 
ER't firtt quettlon and then draws the conversation toward what he/she does. 



#5. The Averkge Talker ' 

4 In thit cite, the average Tnterview will be conducted. E R will be curious, 
cooperative, and talkative. However, ER will have to semi-structure the interview 
to obtain all required Infonr^ation. 




7- 



\ 50 



53 



StopEiflht: Logistics of Fieldworl^ Operations ^ 

■■^ . * 

* ' /" 

In reality, this step is a potpourri of do's and don'ts as you got ready to ontor the field. Thoy 

include for example: 

\ 

\ • Schepuling. f^itetvievvf and formal. observation sessions should be scheduled well in advance 
\ and It it often wise toconfirm the arrangements again shortly before the scheduled time. Scheduling 
\ the intfrvieW or obiervation itself is an opportunity to contact those being interviewed or observed. 
\ For example, over the telephone you can explain the purpose of the case study and then arrange for 
'an appointment. The appointment should then be confirmed in writing with a general estimate of ^ 
\the amount of time needed. Figure 12 presents a sample memorandum for eonfirming an interview 
i(vlth a school. Remember, though, that additional appropriate interviews and observation sessions 
lould come to the fore once in the field and shouJd be followed up. Do not lose valgable oppor 
inKies to collect dat^ becaute the.opportunity is not on the day's schedule. 

The amount of time an actual interview or observation^sslon will last depends upon who Is 
be\ng Interviewed or observed, the purpose of the interview or ok^rvation, and the total amount of 
tirrte available for the case study. Do not forget to schedule an adequate amount of time for traveling 
from one location to another between interviews or observations. Ah interview schedujalike the one 
illustrated in Figure 13 may be helpful as the individual interview and observation schedules are set. 




Selecting Who to /ntem'evy and Whdt to Observe. The selection of observation settings and 
Inten^iewees should be made by the evaluator^^hile people at the site may suggest who the fvaluatort 
shoulo or should not interview or observe, do nljb let them make the final decisions. 



• 'Recording Responses. Whether you take written notes or tape record an interview is up to 
you. However, remember that those being interviewed or observed should be asked if they mind if 
notes are being taken-^egardless of the medium used. 

Alft notes should be very brief-word^ phrases, pONpibly a drawing. Their.purpose is to provide 
stimulation for recall when the notes are vVritten up. This should be done within a matter of hours 
after the data are collected. A single word, even one merely descriptive of the dress of the person, 
or a particular word uttered by someone, usually is enough to 'trip off a string of images that afford 
substantial reconstruction of the observed scene.*^ However, while you paraphrase or summarize 
most of what is obtained through an interview or an observation, do not hesitate to record quotes 
word-for-word. / 

Record all responses and observations as they occur as plainly as possible. Explanatory statements 
may be added as needed. Do not erase any notes since it is less obtrusive and easier to cross out 
incorrect findings. Last, develop a personal shorthand system that will speed up the note-taking. 
Remember, the key to being a good interviewer or obstrver^s to listen and watch attentively-and 
that cannot be done if you are immersed in taking notes. 
■ . ■ .■ 

One final word of warning is appropriate here. Be sure to write ufi each interview or observation 
as soon as possible. If they pile up, do not be surprised to find that several interviews or observations 
get combined or mixed up ip your mind, or that you do not recall details very clearly. 
" -Jf- ■ ' 

• PBitidpatton. Case study evaluators should play a relatively passive role at the site, i.e., while 
they may participate with the people being Ttiterviewed and observed, the interactions should be 
updertaken cautiously. While participation may be good in establishing rapport, it can get In the way 
of observing and interviewing in some circumstances. Bogdan summarizes this trade-off as follows: 



t 9k 
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Figure 12 

Sample Memorandum for 
Confirming Interviews in Schools 

/19 



The " IS one of schools in 

the state of . participating in a study to determine the 

factors affecting^ the plapement of vocatibnal education students in jobs related 

to their training. The study is being conducted by the [ . 

^ , a staff member at the 

is scheduled to interview you. 

> ■ " 

INTERVIEW 

Date ' ' ' Time 

Place. • ; 

Address' ^ " 

Telephone Number , 

.■^ 

The attached sheet provides information about the study. If some emergency • 
prevents you from keeping your appointment, please notify jne as soon as possible 

at . . It is anticipated that the interview will not exceed 

,__B»inMtes. . , ' ' 

■ > ■ » 

« ■ . 

Theinks for your cooperation. We believe this study will be helRfiil to the 

■ . . ' and to others as they work to improve 

vocatioifYAI education programs. 



DATE: 



TO: 



FROM: 



Figure 13 

Daily Interview/ObMrvation Schedule 



I ntarviewer/ObMrver : 

Dayr 



Schedule 



» -a 



a.mVp.m. 
Address: 

Phone: 

a.m. /p.m. 

' Address: 
♦ 

Phone: 
a.m. /p.m. 
Address: 

Phone: 
a.mVp-m. 
Address: 

Phone: 
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Controt your f)articipation, keep in mii^d that youi pr iniai y concei n is 
collfcting data. The amount of participatipn should be such that the 
ratMrcher fits comfortably Into the setting and Is able to establish the 
. kind of rapport he wishes without disturbing the setting or having his 
participation Interfere with his function as an observer. 

• Wfwn in Rome ... It Is Important to learn the language pr jargon that is used at the site. 
This may only mean becoming familiar with a specific set of acronyms, such as NPCTC (Northland 
Park Vocational-Technical Center), or understanding some common words have different meanings 
at that site (e.g., zoo - discipline problem's). The language used will provide insights and clues as to 
how people at the site view and define situations, e^^ent8, and things. 

• Supplias, TherjBisonemorepointthatmustberaised, though, before the data is collected- 
tupplies. A brief "laundry list" of supplies that might be needed should be of help as you conduct 
the case study appears below. While you may not nepd all of these Items, do not forget pens, paper,, 
andmoneyt 



o pens? . 

o note pads? 

o index cards? 

o paper clips? 

o stapler? 

o rubber bands? 



o business cards? 

o recording equipment? 

o maps of city/county? 

o coding systen^? 

o .questionnaires? 

o written description 
of the study? 



o handouts if necessary? 

o credit cards, travelers 
checks, or cash? 

o good humor? 

o aspirin? 

q pleasant disposition? 
o patience? 



Worksheet 8 summarizes the n>ajor points raised in this step. 



.Worksheet 8 
Logistibs of Field work Operation 



Have you:- 




lules 



1) identified and set the interview and obser 
in''advance of entering the site? y 

2) verified all interviews and observations in wrBng? 

3) scheduled the necessary amount of time fok each interview 
jind Obif rvation? 

4) determined who will be Interviewed and what will be observed? 

5) taken notes i|^ the field? 

6) written up your notes daily? 

7) rnaintf ined caution about taking on the role of a participant 
at the site? " 

8) learned the language specific to the site? 

9) determined what supplies will be needed? 



Yes No 



o 

o 

o 
o 

o 
o 

o 
p 



o 
o 

o 
o 
o 
o 

o 
p 
o 



St^pNint: Data Collection , ^ 

As ditcu8sa(i in Chapter IV, cate study data are often collected through systematic observation 
methods-interviewing, observing, and gathering data unobtrusively. This portion of the handbook 
presents soine helpful hints for each method. 

Interviewing. When conducting an interview, it is inriportant to: 

• 'Arrange that a private setting is secured for the intervi^ 

• Explain the degree of confidentiality of responses reported 

• Identify the interviewer and whom the interviewer represents 
a Explain the purpose of t|^e case study — 

a Explain the process by Whichytntervie^s were selected 

a Note the importance of the interview and that you need his/her help 

a Conduct one-on-one rather than a group interview 

a Keep questions brief 

a Save tougher questions for last in the interview 

a Do not breach confidentiality by revealing who said what in a prior Interview 
a Pfnpoint events, people, and places through examples 

a Be prepared, e.g., be on time, have paper and pencil, and know the material to be covered; 
a Be flexible 

a Avoid posing questions in such a way as to receive a socially desirable response 
a Be nonthreatening^ self-control led, polite, and cordial; and 
; a Stay within the alloted time schedule. 

Remember that a good interviewer is a good listener who knows theinaterial to be covered. 
The interviewer makes certain the interviewee answers the questions, keeps track of the information 
covered and to be covered, and pursues information in detail by probing. Figure 14 summarizes 
desirable skills and qualities of an interviewer. , ✓ 
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Figure 14 . 
desirable Skilli and Qualitiet of an Interviewer <^ 

A good Interviewer must have quite an array of personal qualities and interviewing skills 
in relating to people, all of which ultimately facilitate eHcltIng full, relevant, in-depth but 
broad reipontet to the Interview questions. Many of the abilities listed imply great flexibility, 
versatility, and keen judgment of how to handle oneself and the Interaction at each moment. 

In presenting him/herself in the situation and relating to the respondent, the interviewer 
should do the following: 

• Be able to structure his/her role in a realistic, nonthreatening way. 

• Be able to observe a situation while being involved in it. 

• Be at ease but not overly casual, friendly but not too familiar and chatty, curious * 
and investigative but not too nosy or pus*^y. 

• Show sincere interest in the interviewer and respect for his/her feelings and opinions, 
empathizing without becoming Involved. 

• Show curiosity and pleasure in listening without diverting attention to him/her or 
Interjecting his/her own feelings and opinions. 

• Stay neutral and unlnvolved in feelings or issues. 

• Be self-confldeht but not, opinionated, rigid, or moralistic. 

• Be self-aware, seeing himself /herself as others do in the situation and noticing how 
he/she Is functioning. 

• Be able to keep calm in^harged situations, to take rebuffs without flaring back, t6 
tolerate changing moods and divergent opintons while keeping reactions private. 

• Be able to keep perwnal projections at a minimum and yet retain Insights. 

• Be tolerant of a variety of types of people and situations and be able to relate quickly, 
to each situation or person on his/her own terms. 

••Be quick 4o pwcelve the details of a new situation, understand the relationships, grasp 
the terminology, and adjust one's stance and tactics appropriately. This may nrtean 
being more reserved and formal In one situation and more open and Informal m another. 

• Be at ease in en unstructured, open-ended situation yet not allow thf> interview to 
become either a casual visit or a discussion of irrelevant subjects. 

• Be able to shia roles easily as the situation suggests, from active probing to passive 
but attentive listening; from eager, curious questioning to reserved observing. 

• Be able to draw the respondent oUt without manipulating his/her responses In any way. 
\ • Have a keen sense of tlmfng. encouraging respiyes and keeping the '"tf/vlew moving, 

but allowing v»ces of silence for the responded to think and form his/her response. 
— • nCn^ow liow talt to move f rom^perfklal to more personal, sensitive subjects. 

^e Invested In the Inten/lew and the project but keep a sense of balance, objectivity, 
•rid perspective. . 
Have a sense of humor. 
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Flgur* 14 oontlnuMi 

Other Interview characteristics are more Important relative t^ denlincj with the data during 
the interview and later In the report. The interviewer should: 

Be conscientious in recording data and in writing reports. 

• Have clear writing, 

• Have a memory for names and details. 

• Be flexible in recording, alj[e to remember accurately v^en notetak^tlg is inappropriate 
And to reduce Information to brief notes that can stilUbe return^d^o full form in 
reporting. ' ^ ^ 

• Quickly grasp the organization, its relationship and terminology, and the interviewee's 
relationship to it and perspective. 

• Be creative ih framing questions and responses to draw out new material. 
^ Be able to think ahead of the informant 

• Be able to draw out data without manipulating resppnses. 

• Be able to analyze feelings. 

• Be sensitive to possible productive digressions. 

• Seek out wide-ranging implications, think in marfy directions, be creative. 

• Be able to identify rumors and sift them from reliable data. 

' • Be flexible and open to new, unanticipated responses; be unprejudiced anjd follow 
rather than direct the respondent's thinking. ^ 

• Include data that contradict hypotheses and may lead to modifications or rejection 
of them. * 

• Listen to prejudiced, false, or malicious statements while continuing to support the 
speaker. 

• Keep to*the subjed\ in the broadest way; relating data to a theoretical-frameyyork 
without distorting the data. 

• Be able^to keep thiii ultimate research objectives always in mind. , ' 

• Be able to interpret research projects in line with established policy, both in formal 
talks and in informal conversation. 

Finally, ethical considerations require that the interviewer should: 

• Maintain confidences and keep field notes secure. 

• Refrain from criticism and gossip. 

• Refrain from passing judgments on others' behavior or on social situations. 

• Respect others' cuetoms, values, and mannerisms. . / 
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O/^itn^/VV- Wh«n obierving It Is important to: 

• Watch Interactions among people carefully and accurately record thein 

• Study the physical location (e.g.. graffiti on tf\e walls, cleanliness of the building) 

• Be sensitive to the effects the observer may have on ilio obstM vdlioti 

• Keep hii/her biases under tow 

• B« alert to people's body language (e.g.. changes in facial expres&ipn. tone of voice) 
. • Be sensitive to the sequence of events being observed ' 

• Be alert to the expected and the unexpected 

« ■ • 

Gathering Data Unobtrusively. When collecting unobtrusive data, the evaluator should bear 
mind the Items listed above under interviewing and obsen/ing.. If documents are reviewed, it is 
important to: , , t 

• Collect all pertinent documents s 

• Obtain recent as well as older documents that might be pertinent ■ 

• Return borrowed documents when promised » ' -l; ;; 

• Revjew docunSents before you go on site ■ ' 



worksheet 9 repeats the previous information in a checklist fortriat that may be used to 
refresh the evaluator's mennory before entering the site and once on site. 



Worksheet 9 
Data Collection 



Will you be certain to: 



Yes No 



> 

2 ^ 

^ O 

O 

c 

D 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 

12 
13 
14 
15 

16 
17 

^1 
2 
3 
4 
5 
6 
7 
8 

~1 
2 

3 
4 



arrange a private setting for the interview? 

explain the degree of confidentiality of responses reported? 

identify the interviewer? 

identify whom the interviewer represents? 

explain the purpose of the casfe study? 

explain how tlie interviewer was selected? 

note the importance of the interview and that you need his/her help? 
conduct one-on-one rather than a grpup interview? 
keep questions brief? 

save tougher questions for last in the interview? 

do not breach confidentiality by revealing who said what in a 

prior interview? 

pinpoint events, people, and places through examples? 
be prepared? ^ 
be flexible? 

avoid posing questions in such a way as to receive a socially 
desirable response? ( 
be nonthreatening, self-controlled, polite, and cordial? 
stay within the allotted time schedule? 

watch interaction among people carefully? 
record interactions accurately? 
observe the physical location? 

recognize the effects the observer may have on the observation? 
keep biases under tow? 
be alert to body language? 

notice the sequence of events being observed? '^^^ - 

be alert to the^pected and the unexpected? 

collect all apparently pertinent documents? 

obtain recent documents as well as older documents that might ^ 

he_pertinent? 

return borrowed documents when promised? 
review the documents before you go on site? 



o 
o 
o 
o 

o 
o 
o 
o 

C) 

( ) 
o 
o 

o 
o 

o 
o 
o 
o 

C) 

o 
o 
o 



o 
o 

() 
o 
o 
o 
o 

o 

o 

o 
o 
o 

o 
6 
o 
o 
o 
o 

o 



o 



o 
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CHAPTER VII 

THE ANALYSIS, VERIFICATION, AND SYNTHESIS STAGE 



As in any other vcaluation project, the final stage in the case study is the analysis, verification, 
and synthesis of the data. Analysis, verification, and synthesis are undertaken so that the utilization 
qf the report is enhanced. Each of these steps in this stage are described In sonr^e detail below. 



SttpTiii: Analyzing Data 

Analysis of case study data is an ongoing process that begins as soon as the first piece of datum is 
collected. This featurie, i.e., "analysis as you go," distinguishes the case study from other methodologies 
in which the data collection and data analysis are discrete activities. Bogdan states: 

... as the researcher is in ^e field and recording his notes, he then begins 
focusing on certain recurrent themes, which are revealed in observed 
behavior and verbalization. Certain understandings begin to develop and 
sociological concepts are drawn ^pon to malce sense out of the situation. 
Working hypotheses become refined and new concepts are developed. In 
many cases the analysis of the themes direct the observer in his field work 
and help determine the areas in which he will spend his time."^ 

, As you collect and simultaneously analyze the data/*you will find that inferances will be drawn, 
new questions raised, and themes will develop that will adjust the scope, focus,%)d schedule of the 
interviews and observations accordingly. For example, as data are collected from different vantage 
points. It will confirm or contradict the findings that are being obtained, thus raising new questions 
for study. Using the Oneanda example, Infomnatlon about the placement process will be obtained 
through observations artd Interviews with teachers, parents, students, former students, school 
administrators, andjCQmmunity leaders. From their different vantage points, opinions will emer^ 
that will confirm or contradict the patterns of findings that develop. 

Throughout the data collection phase, data is continuously triangulated in order to obtain the 
vast accurate reflection of what is actually occurring at the site. Following the completion of the 
report, the findings may be. tested for their accuracy by having the report reviewed by some of the 
persons at the case study site that were involved In the case study. Alkin et al. describe a verification 
procedure as follows:*^ . . . 

Unlike many field research efforts, the process of review did not end with 
the research tearn. Instead, the case study draft was n^xt circulated to the 
intervlewffs themselves for comment and criticism. Each key informant was 
^ given a copy of the case study draft to examine, and a personal Interview was 
oondMCtid ffter alipwing the informant one or ,tWO weeks to digest the report. 

the int^rvievvs wet^ conducted In a spirit of open exchange of opinion, 
advice, and conitriiotlvt criticism. As might be imagined, the informants did 
not always agree with the draft reports. Their suggestions and comments zeroed 



In on 0^ occMlonal erron of fact dl) convincingly retutod mistaken 

interpretations of events. Other comments were more Subject to dispute; 

where appropriate, further discussions as well as follow-up interviews 

with other informants were conducted to thrash out the issue in question. { 

Thus, we carried out a rather systematic field validation of the authenticity 

ahd accuracy of the case study report. » 

Based upon the additional Information gained in these review sessions, 
the cast study report was edited into final form by the research team. It 
conformed with each Informant's perspective on some points and differed 
on others. All in all, it was what we judged to be fair and accurate in light 
of all the data. 

The nnore^intensive data analysis oGcurs^after all the data are collected and the researchers have 
returned to the office. However, data analysis will probably not begin on the first day back in the 
office since other professional commitments and chores such as typing notes often create a lag 
between collection and analysis. This situation occurs often enough that the case study evaluator 
should be aware of it and plan accordingly. 

However, do not take a long break before the data are analyzed. \% is important that the data 
be analyzed while they are still fresh and clear in the evaluator's mind. Also, after you start to read 
the fidid notes, questions may emerge that require clarification. Needless to say, the longer this 
occurs from the time you were at the site, the greater are the chances that the situation at the site 
has changed or the rapport established at the site has diminished, y 

Generally, the beginning of the data analysis process is to read through all of the data cards 
carefully In order for the evaluator to again become familiar with the information obtained. A second 
reading of the data cards should then be conducted for two purposes. First, during the initial data 
analysis stage (as the data was collected), patterns and themes appeared to emerge. At this point, the 
evaluator should attempt to test the validity of those patterns and themes. Second, when reviewing 
the data cards, look for new patterns and themes. j 

Thfp look at the data by themes using the coding system to pul^ out cards with like pieces of 
Information. Do not hesitate to do some receding at this point by adding some new codes. Remember 
that the coding system repreaents the bnt gu9ss at organizing the data. As such, it is far from the 
goipel and it certainly open to improv«lment. 



The exact procedures used In dat9 analysis vary according to the amount of data collected, the 
Information needs of the audience, and resources available.' For example, some case study reports 
appear at lengthy narratives which rely upon words to describe and explain what occurred at the site. 
Other caie ttudlet rely on content analysis procedures in an effort to quantify the findings or report 
the frequency with which certain things occur. No one specific data analysis technique will serve 9II 
case studies. The technique or techniques chosen, like every other aspect of the case study, must be 
specif ipilly suited to Its needs. Of course, you should have some ideas about how the data will be 
analvlld before It It collected. 

Worksheet 10 Is a brief checklist about data analysis to review the material discussed in this 

•tep. 




Worksheet 10 

Analyzing Data 

Yes No 

1) Will data be analyzed as it is collected? o o 

2) Will an adequate amount of time be allocated for intensive 

data analysis? ^ o 

3) Was thought given to how the data will be analyzed befbre 

' it was collected? « 

4) Would each data card be carefully read for 

a) overall content? *^ *^ 

« b) to test the identified themes? o o 

c) to identify new themes? U o o 

5) Will there be an opportunity to add or to revise the coding 

system in order to make the data more easily retrievable? o o 



Step Eleven: Reporting the Findings 

Reporting the findings fronr an evaluation activity in an accurate, clear, usable way is a 
challenge. Reporting the findings from a case study is no exception. The report should be both a 
descriptive and analytic presentation of the data. The actuar structure of the report should be based 
upon the purpose of the case study evaluation and the interest of the audience. For example, if a 
case study was conducted In ten states which focused on factors that influence the entrance of 
stodents into nontraditional occupations, it could be reported by major themes that appear across 
all sites, by hypotheses that havQ been generated, or by individual states. Each of these three reports 
may be written from the sl^e set of data that was collected and analyzed. It is the presentation of 
^ the findings that differs. The purpose of the case study evaluation and the interests of the audiences 
will determine how the findings should be reported. 

Regardless of the way in which the findings are presented, certain points should be included in 
any case study evaluation report to insure that the reader has enough information about how the' 
case study was qonducted.' These include: 

• Evaluation Purpose. The reader shoujd be informed about the aim of the evaluation effort 
including the rationale behind the study, the major evaluation questions and why they are 
impoltant, and what assumptions (if any) were made. 

• Method. The section should explain why the case study method was chosen, which 
qualitative data collection methods were used and why, and a description of the case 
study procedures. 

V 

• Time a/id Length of the Case Study. The section should report' the dates of the case study, 
^ how much time was spent at each case study site, and so forth. 
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• Sit9s. The tite lelection process should betlearly explained along with the number of 
sites selected and a description of their pertinent characteristicsi Whether or not specific 
sites are identified will depend upon prior arrangements made with each site regarding 
confidentiality. 

• pmitations of the Case Study. At this point, any limitations of the study which might 
have affected the findings should be mentioned. Limitations could include, for example, 
time constraints, elimination of a group of school^ students from the sampling pool, 
lack t>f accessibility to some potential sites which resulted in their elimination from the 
sample, and staffing problems vyhich had an effect on data cqllection or analysis. 

• Relationships at the Case Study Site. The relationships between the evaluators and 
persons at the case study sites should be described in terms of the level of evaluator 
acceptance and any changes in the level of acceptance. 

• Checks on Data. This section should focus on how the evaluators ensured that the data 
collected accurat^y reflected the situation at each case study site. 

Presentation of Findings. This is self-explanatory-let the reader know what the case 
study found. . 

• Conclusions and Recommendations. The conclusions and resulting recommendations 
should be based upon'^he data collected and analyzed. Personal opinions may also be 
included, but they should be labeled clearly as such. 

Both before you begin to write the case study report and after it has been completed, take a 
few minutes to think about the questions posed in Worksheet 11 . 





Worksheet 11 








Reporting the Findings ^ 










Yes 


No 


DidI: 


. . * * 






1) 


Explain the background of the case study (e.g., purpose, limitations)? 


o 


o 


2) 


Describe the site completely? 


o 


o 


3) 


Use accurate, detailed descriptions of actions and behavioi-s? 


o 


o 


. 4) 


Report word-for-word statemer^ts where appropriate? 


o 


o 


?) 


Report whether interaction with informants was effective and 






whether informants were reliable? 




o 


6) 


Work effectively with reliable witnesses and informants? 


o 


. o 


' 7) 


Report on any important trace effects or wear spots? 


o 




8) 


Use documents to add to my observations? 


o 


o , 


9) 


Report' natural and typical conditions? 


o 


o 


10) 


Report whether 1 beanie Involved to the extent that It was 




o <■ 




dliitiptive or unnatural? 


o 


0 ^ 


11) 


Have pre-formed opinions or "^xes to grind"? 


o 

.1 


o . 


12) 


Make lottrpretationi that flow directly from the data? 


o 


o 



Sttp 'l\wtlv«: Utilizing th« Cast Study Findings ^ ^ y 

The extent to which the findings of any case study wllfbo utilized is a question tfiit needs to 
be considered throughout the design, implementation, and reporting of the case study. The literature 
is replete with clairr^ that evaluation information is not used. The case study may be even more 
prone to pitfalls thit wft^ipduce its visibility. Some suggestions for enhancing the probability of 
using case study findings Afe: 

• The people for whom the information is intended shoul^d be involved ih the initial stages 
of planning. They should be involved in clarffying such iskjes as what their evaluation 
information needs ar^ (ahd how the needs are subsequently translated into evaluation 
questions), whether or not a case study is the methodology that will best answer the 
evafuatlon questipns, if adequate resources are available (e.g./ time, staff, money) to 
conduct tKe case study, and wifl the sites selected provide rtieaningful information. 
Although decision-makers will not be involvM in the data collection a|^ analysis phases 
of the case study, their pert in the beginning of the undertaking is crufcial to their eventual 

. use of the findings. 

• The proper audience should receive the case study report. The proper audience includes 
all the stakeholders, i.g., persons who have. an involvement and interest in the^broad 
evaluation question and/or the sites being studied. For example, all persons involved in 
the conceptual portions of the study would be part of the appr'opriate audience as well as 
other parties who express an interest in the study as it is being conducted. >• 

e It IS unlikely that the audience will find the 4ime to read lengthy case study reports. 
Different approaches mey be taken to resolve this problem. For example, it often is useful 
to condense the material into a summary (often known as an executive summary) that ^ 
briefly identifies the' purpose of the study, the evaluators, the methodology and time frame; 
tbe findings, and the conclusions and recommendations. Another method is to extract 
pertlnenrt material from the complete report for peo{j!)le in certain positions, such as teachers, 
counselors, or administrators. If it is important to reach the general public, the findings 
should be condensed and reported in a readable style, perhaps adding photographs and , 
graphic designs. . ^ 

• The use of the report may ohen be enhanced by holding meetings at which the evaluatorv 

^ can report their firldings agd answer questions. This proems Jends fo increase the Jikelihood 

that the case study report will be react. ^ ^ 

The four suggestions above comprise only a partial list. For example, critical to the issue of using 
any evaluation finding is timeliness. If the evaluation information is not available at the time a decision 
is made, it will not be used.^Case study findings are no exceptioni»| 

Each individual situation will have its owribest methods to help increase the use of the data. 
Take a few moments to think' through yo^MpMtuation and the best ways to increase utilization 
of the case study findings— or the finding J^lfnlny other evaluation effort, for that matter. Work- 
sheet 12 in designkl to help ypu thinlc'tlirough jme process. . 



Worksheet 12 
Utilizing the Case Study Findings 



1) Have you looked at your situation and identified the 
best ways to inqrease the utilization of the findings? 

2) Was the- appropriate audience involved in the conceptual 
phase of the case study? ^ 

3) Has the proper audj^mce received the report? 

4) Will different formats of the report be needed for 
different parts of the audience? 

5) Will meetings between the evaluators and each part of 
the audience be scheduled to discuss the report? 



NotM to Chapter VII 



64. Bogdan, Participant Observation in, p. 58. 

65, Alkin et al., Utfng Evaluations, p. 41 
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CHAPTER VIII 
THE CASE STUDY PUZZLE 



This handbook Is designed as an Introduction to a methodology that may be used to evaluate 
vocational education programs/ To make the handbook as user-oriented as possible, the text was 
divided Into throe major parts. Part One briefly reviewed the prck^ess of evaluation, the case Study 
approach, and different data collection methods. In Part Two, the reader was "walked through" twelve 
steps that comprlw the case study process. Part Three consists of a case study puzzle and a aelected 
annotated bibliography. 

Try your hand at the case study puzzle to see how many of the terms introduced In the handbook 
you remember by consulting the clues. T-he terms appear forward, I»a6kward, up, dovim, and diagonally 
-but in all cases they 4orm a straight line. For example, clue 1, interviewing, observing, and documents, 
are examples of gi<ya//t9f/Ve methods. . ' ' \ 

A super sleuth should identify 11 to 15 terms correctly. If you identify between six and ten 
terms it might be a good idea to review the chapters. Less than five terms, well, it is time to start at 
page one again. Good luck. The correct answers appear on the page following the puzzle. 
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pLUES 



' 1 ) Interviewing, observing, documents ... 

2) delineating, obtaining, and providing information for decision making 

3) vtewin^he total picture 

4) If it is not deductive, then it is 

5) a school of psychology 

6) an alternative evaluation methodology 

.7) they try to understand behavior from the person's viewpoint 

8) they seek facts or causes of phenomena 

9) looking, listening, asking 

10) elite or structured 

11) data conducted withoutbeing noticed 

1 2) sites are chosen by identify i ng a [ 

13) collecting data from different sources or by different means 

14) data may be organized as a system 

15) once collected, data should be 

16) audiences should be Involved in early stages to increase the of data 
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ANSWERS TO PUZZLE 



1. qualitative 

2. evaluation 

3. holistic 

' 4. inductive 

5. gestalt 

6, case study 

7, phenomenology 

8. positivists 



9. observe 
10. interview 

unobtrusive 
12. sample 
13» triangulation 

14. coding 

15. analyzed 

16. use 
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CHAPTER IX 
SELECTED ANNOTATED BIBLIOGRAPHY 

Chapter IX contains over thirty five additional resources to which you might refer to enhance 
your knowledge of and skills In naturalistic approaches to evaluation. While the resources selected 
are not a complete listing of those available, they should serwe as a yod place to begin reading. 

Acland, Henry. "Are Randomized Experiments the Cadillacs of Design?" Policy Analysis 5 (2) 
(Spring, 1979):223-241. 

This article discusses the merits of randomized experimental designs as a means to obtain 
informatibn about program effectiveness. USing the emergency school assistance program as 
an example, he argues the need to move past current methodological prescriptions in order to 
. do good research. 

Alkin,'M.C.; Dalllak, R.; and White, P. Using Evaluations. Beverly Hills: Sage Publications, Inc., 
1979.' i 

This "book dlscussesthe utilization of evaluation information and a naturalistic evaluation 
approach. Five case studies of school programs are reported and discussed, examining the 
influence of evaluation upon program decision-making and program operation. The authors 
attempt to explain why and how evaluation did or did not have an impact on program 
decisionmaking. 

Banana, William H. Training In Depth Interviewing. New York: Harper and Row, Publishers, 1971. 

This book Is a training aid for personsJ^am'nQ ^»ow to conduct interviaws or improve their 
interviewing skills by focusing on understandinjg the interaction process occurring in an inter- 
view. It describes a series of detailed guides for each phase of the Interview process: planning; 
setting up a practice interview; summarizing tbe data; content and process analysis; and 
evaluating interviewing skills. Interpersonal theory and specific evaluation of interview behaviors 
is interwoven, through the book. An interactional approach is used; emphasizing objective, 
analytical, and subjective sensitivity skills related to the interviewer and to the interviewee; 
and the impact these have on the other. An annotated bibliography is included at the end of 
the book. 

Bodgan, Robert. Participent Observation In Organizational Settinffi. Syracuse: Syracuse University 
Press, 1972. 

The reader 11 introduced to participant observation and the stages ifi this type of research: the 
pre-fieldvtmrk stage, the fleldwork stage, and the intensive analysis stage. The discussions in 
each stage focus on "how to" conduct participant pytervations. 



Bodgan, Robert, and Taylor, Steven J. Introduction to Qualitative Research Methods: A 

. Phanomenological Approaqh to the Social Sciences. New York: John Wiley and Sons, 1975. 

This book deals with the sociological methods and techniques of qualitative research. It is 
presented in two parts, plus field notes and an extensive bibliography. The Introduction deals 
with qualitative methods in general and with a number of theoretical underpinnings such as 
ethnomethodology. Part One concentrates on how to conduct qualitative research, pre-fieldwork 
activities, collecting data, and analyzing data. Part Two discutsas the writing up and presentation 
of fir^dingt and illustrates ways in which research findings can best be presented. 

Bradburn, N.M.; Sudman, S.; and Associates. Improving Interview Method and Questionnaire Design. 
San Francisco: Jossey Bass Publishers, 1979. 

This publication (from the National Opinion Research Center Series in Social Research) addresses 
the reluctance of many people to give full and honest answers when asked about such threatening 
topics as sexual behavior, bankruptcy, criminal arrests, or alcohol consumption. It reports findings 
from a series of experimental studies and presents detailed information on the amount of distor- 
tion that occurs with questions of varying degrees of threat and with different survey procedures. 
Also, it outlines practical techniques for increasing respondents' willingness to cooperate and for 
reducing their misreporting of behavior; and enables survey users to critically evaluate survey 
results and identify potential sources of error. ^ 

Cook, Thomas D., and Reichardt, Charles S. (eds.). Qualitative and Quantitative Meth^yds in Evaluation 
Research. Beverly Hills: Sage Publications, Inc., 1979. 

Eight papers comprise this collection which focus on qualitative versus quantitative methods, 
the need for qualitative methods in evaluation research, degrees of freedom and the case study, • 
reconciling qualitative and quantitative data, a rapprochement of quantitative and qualitative 
/ methods, use of photographs, ethnographic contributions to evaluation research, and linking 
process and impact analysis. 

Denzin, Norman K. Sociological Methods: A Sourcebook, Chicago: Aldine Publishing Company, 1970. 

This book consists of a series of articles concerned with the problems related to developing 
theory and implementing valid inquiry. Specifically, it discusses empirical research from a 
naturalistic perspective. The nature of social inquiry, sampling technique, measurement, inter- 
views, design and analysis, and social experiments are raised. SOrvay. participant observation, 
life^histories, unobtrusive measures, triangulation and contingencies, £H3d problems in the 
.^ecution of social research are^also discussed. \^ 

Denzin, Norman K. "The Logic of Naturalistic Inquiry." S&c/a/ Forces 50 (1971): 166-182. 

One form of empirical inquiry, naturalistic behaviorism, is presented and compared to other^ 
theories in this article. A framework for organizing naturalistic studies also is provided. Special 
attention is given to show how the investigator obtains samples; selects and uses naturalistic 
induction; analyzes various case types; provides explanatory models of the interaction; and 
records and analyzes his/her own behavior. 

Dexter, Lewis A. El/lnf and Specialized Interviewing. Evanston: Northwestern University Press, 1970. 

This book is designed for social scientists engaged in the collection of information through 
'elite' interviewing. The author describes an elite Interview as "the giving of special, non- 
standardized treatment'' to the interviewee. It describes how interviews can be conducted, the 
ways in which questions can be posed, ways replies can be recorded, and the metliod of drawing 



78 



78 



infertncM. Particular topics deal with: Interview procedure; oral history interviewing; the 
kind of truth one can expect to get from an interview; and an assessment of interviewee's 
responses and reactions. An annotated bibliography is included. 



Douglas, Jack D. Inv^tigative Social Research: Individual and Team Field Research. Beverly Hills; 
Sage Publications, Inc., 1976*. 

How different methods of social research are used to obtain different types of information with 
various types of social groups is discussed. He claims that the selection of the methodology 
largely is chosen by how we view the environment. A major portion of this book describes how 
the major forms of natural involvement and field research methods are used to gain appropriate 
information. Attention is also given to the problems society poses for field researchers in their 
studies^ particularly xhcm related to evasiveness, secrecy, deceit, frontwofk, and conflicts, plus 
ways in which can be circumvented. 

Eisner, Elliott T. The Perceptive Eye: Toward the Reformation of Educational Evaluation. Stanford: 
Stanford Evaluation Consortium, Department of Education, Stanford University, 1975. ^ 
(ED 128 408). 

It is suggested that a supplement to the use Qf scientific procedures of describing, interpreting, 
and evaluating educational settings is required rather than ^purely scientific approach. Such a 
new, non-scientific approach can be applied, providing that it incorporates 'educational con- 
nolsseurship' and 'educational criticism'. The paper defines the meaning of those terms and 
describes the ways in which each can be used to evaluate educational settings. 

Fehrenbacher, H.L.; Owens, T.R.; and Haenn, J.F. The Use of Student Case Study Methodology In 
Program Evaluation. Research Evaluation Development Paper Series No. 10. Portland, Oregon: 
Northwest Regional Educational Laboratory, 1978. 



This paper discusses the case study method, o^es of the case study, and provides specific 
procedures for ciinducting a case study. The latter portion illustrates the uses of the case study 
method for evaluating an Experience-Based Career Education (EBCE) program and contains an 
illustrative case study. 

Fiedler, Judith. Field Research: A Manual for Logistics and Management of Scientific Studies in 
Natural Settings. San Francisco: Jossey-Bass Publishers, 1978.- 

This general introduction to field research outlines the methodology associated- with the design, 
planning, ancj management of investigations of real-life settings. Fiedler discusses .techniques 
that facilitate fieldwork and offers a rationale for determining the needs for and utilization of 
resources in the field. Each stage of field research management is discussed. These include site 
relations, communication, field staff management, cost estimates, budgeting, supplies, and 
documentation. » , 

Guba, Egon'G. Toward a Methodology of Naturalistic Inquiry in Educational Evaluation. CSE Mono- 
graph Series in Evaluation, No. 8. Los Angles: Center for the Study of Evaluation, University 
of California, 1978. 

The nature of naturalistic inquiry, vvays in which it differs from convi|itional inquiry, hovy it is 
similar to other contemporary methodologies, ^f\d the need for this type of research are pi||iente( 
in the first taction of thb monograph. The second sectioh discussm refactions to conventiorial 
inquiry methodologies and the development of models based uaon and compatible with natural- 
istic Inquiry concepts. The third sectio/i concerns itself with th(ree major methodological problem 
\ that should be considered by those wishing to use this type of approach; defining the boundary, 
with focusing and with assuring authenticity. 
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Hamilton, D.; MacDonald, B.; King, C,; Jenkins, D.; and Parlett, M. (eds.). Beyond the Numbers 
Game: A Reader in Educational Ei/aluation. Berkeley, California: McCutchen Publishing 
Corporation, 1977. ^ 

This Is a collection of roadlngs r^lntod to alternative or illuminative (nontraditional) nnodos of 
evaluation. Section one discussed this approach in reference to the study of innovative proyrams. 
Section two covers the process of curriculum developnrient, particularly with respect to choosing 
and formulating educational objectives and organizing learning objectives around the educational 
objectives. The third section presents alternative ways of evaluating educational programs by 
five experts in the field. A series of papers on alternative methodologies are compiled in the 
fourth section. These describe 'illuminative' evaluation approaches which are 'adaptive' and 
'responsive'. The emphasis is on the methodological aspects associated with those used in social 
ar>thropology. Section five attempts to answer the question of whether illuminative studies are 
more useful, informative, or profound than traditional forms when the earlier theories described f 
in the book are put into practice. A series of sample reports, using different ftyles and types of 
evaluation procedures, are given to show the method and theory in action. 

House, Ernest R. The Logic of Evaluative Argument CSE Monograph Series in Education, No. 7. 
" Los Angeles: Center for the Study of Evaluation, University of California, 1977. 

This monograph provides a theoretical analysis of evaluation. The first section discusses evalu- 
ation as an argument. The equivocality of evidence, evaluation as an art of persuasion, recipients 
of information, premises of agreement quantitative and qualitative issues, and the development 
of an argument are covered in the first section. The second section covers the logic of argument, 
exemplary studies, naturalistic evaluation, and objectivity, validity, and impartiality issuer 

Kester, Ralph J. Using Systematic Observation Techniques in Evaluating Career Education. Columbus, 
Ohio: The National Center for Research in Vocational Education, 1979. 

This handbook Introduces the reader to three systematic observation techniques-observing, 
interviewing, and gathering data unobtrusively-that may be used to evaluate career education 
programs. It provides practical guidelines on how to use each technique; things to consider 
when using the techniqueli; how to organize, analyze, and justify the data; and how to enhance 
systematic observation skills. 

Jason, Martin H., and Schwartz, (Henrietta S. A Gtfjffebook to Action Research for the Occupational 
Educator. D^epartment of Adult, Vocational and Technical Education, State Board of Education, 
Illinois Office of Education, 1976. 

This guidebook on conducting action research arose out of an action research workshop con- 
ducted for occupational educators in Chicago high schpols; Various methods of conducting 
research with examples applicable to vocational education are given throughout the text 
Step-by-step procedures are presented covering such topics as setting research hypotheses, 
sapipling procedures, testing hypotheses, research design, evaluating curriculum, survey research, 
the case study, anthropolc^fcal and ethnographic research, af% preparing the research report. 
This practical guide to applied educational research methodology shoul?) be of considerable 
help to teachers, counselors, and administrators. . 

Johnson, John M. Doing Field Research. New York: The Free Press, 1975.^ 

This book provides a detailjsd description and analysis of sociological field research methods. 
It is concerned with outlining how investigations are carried out with groups in particular social 
settings. The first section of the book discusses the major arguments in the traditional methodo- 
logical literature about the problems of conducting field research. The second section describes 
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some of the more applied prpblem^, namely: ways of gaining and managing entry; methods 
used In developing personal relations and trust; personal relationships in field research and 
their influence on data collection; and ways of constructing a national account of field research. 
The third section compares the traditiortai cor)ceptions of field lesearch practice and the author's 
experience within the context of current debates about social science objectivity. 

McCall, George J., and Simmons, J.L (eds,), Issues in Participant Observation: A Text and Reader. 
Reading, Massachusetts: Addison Wesley Publishing Company, 1969. 

This analytical review of methodological issues brings together in one volume a range of selected 
articles covering eight key areas. They concern: the nature of participant observation; field 
relatioos; data collection, recording, and retrieval; the quality of data; generation of hypotheses; 
the evaluation of hypotheses; publication of results; and comparison of methods. ^ 

♦ 0 

Patton, Michael Quinn. Utilization-Focused Evaluation. Beverly Hills: Sage Publications, Inc., 1978. 

This is bclsed upon the results of experiences gained from: studies of utilization; conducting 
evaluation studies; current theories of formal organization and organizational dynamics; decision- 
making theory and policy analysis; and evaluation research literature. It attempts to provide a 
comprehensive and pragmatic approach to program assessment by means of an overall framework. 
Each chapter develops further aspects of this theme. The definition of utilization of evaluation, 
the processes of utilization With respect to decision making practices; focusing and framing 
evaluation questions; goal clarification; implementing the evaluation study; alternative paradigms 
of evaluation measurement and design; and the analysis, interpretations, disseminatioii, and 
utilization of the finding^ are some of the themes discussed. 

Patton, Michael Quinn. Qualitative Evaluation Methods. Beverly Hills: Sage Publications, Inc., 1980. 

This book emphasizes strategies for generating valid, useful and credible qualitative information 
for decisron-making. Divided into three parts, it focuses on different approaches to f feld obser* 
vation, interviewing, and data analysis. It also includes examples of different research strategies, 
samples of qualitative instruments, and excerpts from reports. 

Rist, Ray C. ''Blitzkrieg Ethnography: On the Transformation of a Method into a Movement". 
Educational Researcher 9{2) (February, 1980): 8-tO. 

This article discusses the increased popularity of ethnography in respect to the costs and benefits 
that accrue. For example, as costs have stemmed from education's over-reliance on quantitative 
methods when they were "appropriate or unable to answer the questions at hand. So also 
qualitativdresearch f^es growing costs. The more reliance on the method as an end in itself, 
the less it Is a meaningful research tool." 

Schatzman, Leonard, and Strauss, Ansel m L. Field Research: Strategies for a Natural Sociology. 
Englewood Cliffs, New Jersey: Prentice- Hal I, Inc., 1973. 

The principal focus of this book is on field researchers. It describes the way in which researchers 
interact and deal with the field situations, the problems they face, and the options that are open 
to them. Methodological and philosophical issues relating to field research precede a series of 
strategies concerning: entering an organization, watching, listening, recording, analyzing, and 
comnfiunicating. Instrumentation for measuring the intervening mechanisms, research design, and 
ethics are some^of the issues covered in the epilogue. 
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Sechrest,. Lo« {Gil). Unobtrusive Measurement Today. Sim frmChco: Jossoy B;jss Publishers, 1979. 

This source t)ook is « collection of six ai tides denliiuj with unobtrusive ii'uMsuies. After art 
overview of the prorniso of unobtrusive riuvmires. the mithors discuss the use of direct tibserva 
tions: how to dosiyn uno'i)trusive field experiments; a reconciliation f)etween unof)trusive aruf 
cross-cultural research approaches; nonverbal behaviors as unobtrusive riieasures; and trace 
measures. 

Siedman, Eileen. "Why Not Qualitative Analysis?" Public Administration^eview. (July/Autjust, 
1977). 415 417. 

The author argue? that qualitative analysis, iti contrast to quantitative analysis, can be used to 
restore the legitimacy of subjectivity and give cretfibility weight so that docfsjons and ^ 
actions can be aqcurately assessed. Program analysis, she claims, is an art which incorporates 
both qualitative and quantitative nieasures. Program success should be determined by comparing 
legislative intent to the results as viewed by tfie reciffient. That is,'i)y assessing "reality", rather 
than just analyzing information. 

Stake, Robert E. "The Case Study Method in Social Inquiry." Educational Researcher. 7(1978): 5- 8. ' 

In this article, the role of the case study method in social inquiry is discussed. He argyes that ^ . 
the case study will continue to be a useful means of exploration for those who are searching 
for explanatory laws. He posits that ". , . because of the universality and importance of fexperi 
ential understanding, and because of their compatibility with such understanding, case studies 
/;an be expected to continue to have an epistemological advantage over other inquiry methods 
as a basfs for naturalistic generalization". 

Tikunoff, William J., and Ward, Beatrice A. (eds.). "Exploring Qualitative/Quantitative Research 
Methodologies in ^6ucaWor\." Anthropology and Education Quarterly, VIII (1977):37-163. 

This issue consists of eleven articles and nine critiques which provide an overview of research 
methodologies, the next steps in quaUtative data collection, assessing language development, 
. why do-demonstrations?, how to identify effective teaching, and assessing rape relations in the 
, classroom, ' ' 

Van Maaaen, Uohn (ed.). "Qualitative Methodology," Administrative Science QuarWly. 24 0979): 
519-702. . ^ 

- This4ssul^of ASQ highlights qualitative methodology. Pourtefen articles aldress suc^ tof^ics as 
■' ethnography organizational ethnography, qualitative research techniques in economics, data 
analysis, triangulation, surveying, quantitative versus qualitative data, and unobtrusive measures. 

Wax, Rosalie H. Doing Fieldwork: Warnings and Advice. Chicago: University of Chicago Press, 1971. , 

Theoretical presuppositions of fieldwork are covered in the first chapter of this book. She 
- " stresses the imjbortance of the investigator obtaining an insider's view of the culture and that . 
this involves becoming 'socialized' so that a more comprehensive understanding can be gained. ^ 
Such understanding is considered ah essential precondition of research and enables a more 
adequate explanation to be made. Qther chapters in the introductory section describe some of 
the difficulties that may be encountered during the resocialization process, some interesting 
social research conducted In earlier times in Europe and America, and the difficulties and 
ambiguities encountered while In the field. The remaining portion of the bopk reports on three 
malor case studies: fieldwork in the Japanese American relocation centers; fieldwork on the 
Thrashing Buffalo reservation; and fieldwork among the Six Friendly Tribes. The autobiographical 
materials are Intended as case studies in the training of future fleldworkers. J 
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Webb, EJ.; CamDbell, D.T.; Schwartz, R.D.; and Sechrest, L Unobtrusive Measures: Nonreactive 
iResearch fnwht> Socia/ Scfffnces. New York: Ranrd McNaMy, 1^6, 

This bqo|^ describes and encourages the use Of a w^de range of unobtrusive measures for 
gathering re8|^reh data. The authors recommend the use of supplementary methods for 
testing the san\6 social science variable or, as they descjribe the process, . . [researchers 
should use] multiple opefationism, a collection of methods combined to avoid sharing 
same weaknesyis." They claim that such alternative approaches are designed to supplement 
and cross- validate the interview process. Various nonreactive measures are discussed, such as: 
the s<ilectiv6 erosion of titles; the*accretion rate of whiskey consumption; the shrinking 
, diameter of a circle of children; pupil dilation; library withdrawa!3;^istortion in the sfze of 

tei-tain drawings; and the/degree of selective clustering in lecture halls. / — 

Willem^ Edwtn P., and Raush, Harold L. {eds.). Naturalistic Viewpoint in Psychological Research. 
New York: HoU, Rinshart, and Winston; Inc*, 1969. / * 

This is'a collection of ten papers e)tpressing different viewpoints related to naturafistic research, 
the main themtf" is concerned with, developing a methodology of naturalistic inquiry, rathfer 
than a discussion of philpsopl^ical issues and specific techniques per se, although those are 
. discussed tasome degree^ Three broad ilriethodological flreas are presented in the next ten 
' chapters. The f ir^it of'the three discUss rnore general aspects of ecologicdl research and nitural- ^ 
jstic methodology. The authors argue that there is ar> urgent need 4pr expansion, research, and 
devplopm'ent of irr1f)roved techniques for observation^d analysis in tb^behavjcfral sciences. 
\ A number of criteria that can be considered also is presented. The secortd area concerns itself . 
with establishing more dearly defined sections witWn naturalistic research, namely: ,the study 
of subhuman primates; clinical psychology; resiMrch on attitudes and opinions; and th"^ cultural 
' • study of personality* In section\three, the emphasis on developing an eoologiqjal theory 6f 

intervention by discussing how individuals becopr^e .effective and survive in varied sociaLerwiron- 
\ ^ mertts. The concluding section summarizes the is^O^i and highlights the Unsolved pro^l^ml 

VVillit;'<Seorge (ed.h Quatitative Evaluation: Con\:epj^ and Cdses in Currfculurn Evaluation. Berkeley: 
McCutchen Publishing Corporation, 1978. ' ' \ " 

This tx)bk attempts to link together significant practical and ttieoretical examples of qualitative 
approaches to'ttvalUation. Topics about such'lEi0pi;oaches are raised >£rom a phenomenoJogiCal 
' perspective and relate particularly to the evaluation of classroorp interactions. T^fifrl^obk 
• > ^discusses how qualitative research r^sts upon quite differfent assumptions. Uses different tech* 
niques, and 3 variety of criteria uncommon to thJk of conventional rpjearch methords.^ A Stfri6s^ 
of case studies are presented which describe how ilfc researt:^rs can become involved directly 
. in'the educatronal p^bcess; be'se>)sUive to^the eventstaRingrttlace; operate (iv^intfie objectives 
of the 5tudy; and be able to synthesize and report the evewfl clearly, accmM Objec- 
tively .^Tfiq.bcxjk iS'divided into three sections, "jjte firs^section descVibMne theory ;behmd 
qiialitatiye evaluation; lK selection of exemplar/^ase studies, including flHscussion of the 
method Dsed iiY each case, are presented in the second section. 'An overvie^ of the m^jor issues 
, . presented in t[i'e case studies is p'rdsent^ in the third s^ctibn. 

Wilson/ Sflke. ''Exploration of the IbelulVless o\ Casfe Study Byfi\uht\pnsi'\€valuation Qumer/y 
. 30979): 446T4fe9; * , * ^V>, ^ ^ ^ , 

k ■ _ ^ i> * ■ ^ . ■ ^ ■ a . 

. Wilson ej^amine* (S'ltically \he usef ulness^of the case study afiprpach to educators,«nd parent, 

groups; Pgrticutar attention is p*id to the defiriltion of c^^e studiels, the relevance this approach 
; is,ciaimec!l, to have for practitioners, and issues which influence Us usefulness. Limitations relating 
to the tinrte to read .and the Ien9^ of the reparts'as w^ll as {he Values and ^eoretical orientations 
pffhgF^aders are said to det^rmirre the degrpe to which the findings wiH^ beaccepted, can be . 
gertdliized, and will" bf, used in ddcTsion-rnakin^^ , * , v ,^ * . - 

, , , ■ ■ ■ r ■ ' ^/ ".-■i-- V ^ ' J ■ . • ' ■ ' ; ' • ■ " . 



Wilson; Stephen et al. The Use of Ethnography in Educational Evaluation. Chicago: Center for 
New Schools, 1974. u 

The author discusses the use of rnGthodplogy and firMings using ethnography in evaluating an 
Innovative educational proqfam. A description is given of the evaluation model, used to clarify 
the'context In which ethn<4raphy is used. The use of ethnography is also discussed from two 

• perspectives: problem* arising from the discrepancy between the assumption and processes of 
ethnography and those of evaluation; and problems arising during the implementation of the 

^ evaluation. 

Wolf R L •■ Andis M.F.; Tisdal, G.E.; and Tymitz, B.L. New Perspectives in Evaluating Museum 
' Environrmh'ts: An Annotated Bibliography. Department of Psychological Studies, Office of 
Museum Programs, Smithsonian Institution, 1979. 
r This annotated bibliography outlines evaluation methods that are relevant to a museum context. 
.T^bibliographyshas been divided into four sections, each having a brief explanation of the 
(JBre of the books and articles included and their relevance to naturalistic museum evaluation, 
piese sect«)ns concern: background and rati9nale; collection strategies; analysitf:strategies; and 
sample studies and reports. 

wolf Robert L.'; and Tymitz, Barbara Ca Preliminary Guide for Conducting Naturalistic Evaluation 
"in Studying Museum Environments. Department of Psychological Studies, Office of Museum 
Programs,' Smithsonian Institution (n.d.). ' . , 

• This is a guide describing naturalistic evaluation procedures applied to museum settings. The 
first of a series it provides an overview of naturalistic evaluation procedures and describes a 
range of interview strategies with the goal 0f making evaluators more sensitive.to the subtleties 
of perceiving and understanding the environment. 
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EVALUATION HANOPOOKS SERIES 

• Guicjlelines and Practices for Follow-up Studies of Former .Vocational Education Students, 

• Guidelines and Practices for Follow-up Studied of Special Populations 
The Case Study Method: Guidelines, Practices,. and Appliqatlons for Vbcatidnal Education 

• Performance Testing: Issues Facing Vocational EducatitHi 

• Evaluation Guidelines and Practicejs for State Advisory Councils 



CAREER EDUCATION MEASUREMENT SERIES 

e Assessing Experiential Learning in Career Education 

• Career Education: A Compendium of Evaluation Instruments 

• Improving the Accountability of Career Education Programs: Evaluation Guidelines 
and Checklists 

• A Guide for Improving Locally Developed Career Education Measures 
Using Systematic Observation Techniques in Evaluating Career Education 



' VOCATlONAl\ EDUCATION OUTCORAES SERIES 

• ViQ^oints on I nterpretir^ Outcome Measures in Vocational Education 

• Vocational Education Measures:^lnsfrumenis to Survey Former Students and Their Employers 

• Vbeational Education Outcorhes: An Evaluative Bibliography for Empirical Studies 

• Vocational EdCication Outcomes: Perspective fdr Evaluation 
Vocational Education Outcomes: A Thesaurus of Outcome Questions 

• Vocational Education^Outcomes: Annotated Bibliography of Related Literature 

• $ome Key Outcomes of Vociertional Education: A Report on-^jEydluation Criteria, Standards, 
and Procedures 



For Futihsr Information . ^ ^ ^ 

For information concerning the abqye publications, please contact: 
Program Information Office \ 

The National Center foKReMirchwp Vocatiortal Education 
. * The Ohio State Uniyersity ^ ' 
- 196J0 Kenny Road J 

^ . CdlumJiHJS, Ohio 432rp ^ \ 

] • (614) 486-3665 v ' 
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